You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2018/01/07 23:47:00 UTC

[jira] [Assigned] (SPARK-22983) Don't push filters beneath aggregates with empty grouping expressions

     [ https://issues.apache.org/jira/browse/SPARK-22983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Apache Spark reassigned SPARK-22983:
------------------------------------

    Assignee: Josh Rosen  (was: Apache Spark)

> Don't push filters beneath aggregates with empty grouping expressions
> ---------------------------------------------------------------------
>
>                 Key: SPARK-22983
>                 URL: https://issues.apache.org/jira/browse/SPARK-22983
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.1.0, 2.2.0, 2.3.0
>            Reporter: Josh Rosen
>            Assignee: Josh Rosen
>            Priority: Critical
>              Labels: correctness
>
> The following SQL query should return zero rows, but in Spark it actually returns one row:
> {code}
> SELECT 1 from (
>   SELECT 1 AS z,
>   MIN(a.x)
>   FROM (select 1 as x) a
>   WHERE false
> ) b
> where b.z != b.z
> {code}
> The problem stems from the `PushDownPredicate` rule: when this rule encounters a filter on top of an Aggregate operator, e.g. `Filter(Agg(...))`, it removes the original filter and adds a new filter onto Aggregate's child, e.g. `Agg(Filter(...))`. This is often okay, but the case above is a counterexample: because there is no explicit `GROUP BY`, we are implicitly computing a global aggregate over the entire table so the original filter was not acting like a `HAVING` clause filtering the number of groups: if we push this filter then it fails to actually reduce the cardinality of the Aggregate output, leading to the wrong answer.
> A simple fix is to never push down filters beneath aggregates when there are no grouping expressions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org