You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Weichen Xu (JIRA)" <ji...@apache.org> on 2019/08/05 10:07:00 UTC
[jira] [Updated] (SPARK-28621) CheckCartesianProducts do not work
correctly
[ https://issues.apache.org/jira/browse/SPARK-28621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Weichen Xu updated SPARK-28621:
-------------------------------
Description:
CheckCartesianProducts do not work correctly.
There're some cases:
providing:
{code}
spark.range(2).createOrReplaceTempView("sm1") // can be broadcast
spark.range(50000000).createOrReplaceTempView("bg1") // cannot be broadcast
spark.range(60000000).createOrReplaceTempView("bg2") // cannot be broadcast
{code}
and suppose `spark.sql.crossJoin.enabled=false` (by default)
1) Some join could be convert to broadcast nested loop join, but CheckCartesianProducts raise error. e.g.
{code}
select sm1.id, bg1.id from bg1 join sm1 where sm1.id < bg1.id
{code}
2) Some join will run by CartesianJoin but CheckCartesianProducts DO NOT raise error. e.g.
{code}
select bg1.id, bg2.id from bg1 join bg2 where bg1.id < bg2.id
{code}
was:
CheckCartesianProducts do not work correctly.
There're some cases:
providing:
{code}
spark.range(2).createOrReplaceTempView("sm1") // can be broadcast
spark.range(50000000).createOrReplaceTempView("bg1") // cannot be broadcast
spark.range(60000000).createOrReplaceTempView("bg2") // cannot be broadcast
{code}
and suppose `spark.sql.crossJoin.enabled=false` (by default)
1) Some join could be convert to broadcast nested loop join, but CheckCartesianProducts raise error. e.g.
{code}
select sm1.id, bg1.id from bg1 join sm1 where sm1.id < bg1.id
{code}
2) Some join will run by CartesianJoin but CheckCartesianProducts DO NOT raise error. e.g.
{code}
select sm1.id, bg1.id from bg1 join sm1 where sm1.id < bg1.id
{code}
> CheckCartesianProducts do not work correctly
> --------------------------------------------
>
> Key: SPARK-28621
> URL: https://issues.apache.org/jira/browse/SPARK-28621
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.4.3
> Reporter: Weichen Xu
> Priority: Major
>
> CheckCartesianProducts do not work correctly.
> There're some cases:
> providing:
> {code}
> spark.range(2).createOrReplaceTempView("sm1") // can be broadcast
> spark.range(50000000).createOrReplaceTempView("bg1") // cannot be broadcast
> spark.range(60000000).createOrReplaceTempView("bg2") // cannot be broadcast
> {code}
> and suppose `spark.sql.crossJoin.enabled=false` (by default)
> 1) Some join could be convert to broadcast nested loop join, but CheckCartesianProducts raise error. e.g.
> {code}
> select sm1.id, bg1.id from bg1 join sm1 where sm1.id < bg1.id
> {code}
> 2) Some join will run by CartesianJoin but CheckCartesianProducts DO NOT raise error. e.g.
> {code}
> select bg1.id, bg2.id from bg1 join bg2 where bg1.id < bg2.id
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org