You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Josh Rosen (JIRA)" <ji...@apache.org> on 2019/06/25 21:16:00 UTC
[jira] [Updated] (SPARK-28166) Query optimization for symmetric
difference / disjunctive union of Datasets
[ https://issues.apache.org/jira/browse/SPARK-28166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Rosen updated SPARK-28166:
-------------------------------
Description:
The *symmetric difference* (a.k.a. *disjunctive union*) of two sets is their set union minus their set intersection: it returns tuples which are in only one of the sets and omits tuples which are present in both sets (see [https://en.wikipedia.org/wiki/Symmetric_difference]).
With the Datasets API, we can express this as either
{code:java}
a.union(b).except(a.intersect(b)){code}
or
{code:java}
a.except(b).union(b.except(a)){code}
Spark currently plan this query with two joins. However, it may be more efficient to represent this as a full outer join followed by a filter and a distinct (and, depending on the number of duplicates, we might want to push additional distinct clauses beneath the join, but I think that's a separate optimization). It would cool if the optimizer could automatically perform this rewrite.
This is a very low priority: I'm filing this ticket mostly for tracking / reference purposes (so searches for 'symmetric difference' turn up something useful in Spark's JIRA).
was:
The *symmetric difference* (a.k.a. *disjunctive union*) of two sets is their set union minus their set intersection: it returns tuples which are in only one of the sets and omits tuples which are present in both sets (see [https://en.wikipedia.org/wiki/Symmetric_difference]).
With the Datasets API, we can express this as either
{code:java}
a.union(b).except(a.intersect(b)){code}
or
{code:java}
a.except(b).union(b.except(a)){code}
Spark currently plan this query with two joins. However, it may be more efficient to represent this as a full outer join followed by a filter and a distinct (and, depending on the number of duplicates, we might want to push additional distinct clauses beneath the join, but I think that's a separate optimization). It would cool if the optimizer could automatically perform this rewrite.
This is a pretty low priority: I'm filing this ticket mostly for tracking / reference purposes (so searches for 'symmetric difference' turn up something useful in Spark's JIRA).
> Query optimization for symmetric difference / disjunctive union of Datasets
> ---------------------------------------------------------------------------
>
> Key: SPARK-28166
> URL: https://issues.apache.org/jira/browse/SPARK-28166
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Josh Rosen
> Priority: Minor
>
> The *symmetric difference* (a.k.a. *disjunctive union*) of two sets is their set union minus their set intersection: it returns tuples which are in only one of the sets and omits tuples which are present in both sets (see [https://en.wikipedia.org/wiki/Symmetric_difference]).
> With the Datasets API, we can express this as either
> {code:java}
> a.union(b).except(a.intersect(b)){code}
> or
> {code:java}
> a.except(b).union(b.except(a)){code}
> Spark currently plan this query with two joins. However, it may be more efficient to represent this as a full outer join followed by a filter and a distinct (and, depending on the number of duplicates, we might want to push additional distinct clauses beneath the join, but I think that's a separate optimization). It would cool if the optimizer could automatically perform this rewrite.
> This is a very low priority: I'm filing this ticket mostly for tracking / reference purposes (so searches for 'symmetric difference' turn up something useful in Spark's JIRA).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org