You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (Jira)" <ji...@apache.org> on 2022/11/09 01:44:00 UTC
[jira] [Resolved] (SPARK-41035) Incorrect results or NPE when a literal is reused across distinct aggregations
[ https://issues.apache.org/jira/browse/SPARK-41035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-41035.
----------------------------------
Fix Version/s: 3.3.2
3.2.3
3.4.0
Resolution: Fixed
Issue resolved by pull request 38565
[https://github.com/apache/spark/pull/38565]
> Incorrect results or NPE when a literal is reused across distinct aggregations
> ------------------------------------------------------------------------------
>
> Key: SPARK-41035
> URL: https://issues.apache.org/jira/browse/SPARK-41035
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.2.2, 3.4.0, 3.3.1
> Reporter: Bruce Robbins
> Assignee: Bruce Robbins
> Priority: Major
> Labels: correctness
> Fix For: 3.3.2, 3.2.3, 3.4.0
>
>
> This query produces incorrect results:
> {noformat}
> select a, count(distinct 100) as cnt1, count(distinct b, 100) as cnt2
> from values (1, 2), (4, 5) as data(a, b)
> group by a;
> +---+----+----+
> |a |cnt1|cnt2|
> +---+----+----+
> |1 |1 |0 |
> |4 |1 |0 |
> +---+----+----+
> {noformat}
> The values for {{cnt2}} should be 1 and 1 (not 0 and 0).
> If you change the literal used in the first aggregate function, the second aggregate function now works correctly:
> {noformat}
> select a, count(distinct 101) as cnt1, count(distinct b, 100) as cnt2
> from values (1, 2), (4, 5) as data(a, b)
> group by a;
> +---+----+----+
> |a |cnt1|cnt2|
> +---+----+----+
> |1 |1 |1 |
> |4 |1 |1 |
> +---+----+----+
> {noformat}
> The same bug causes the following query to get a NullPointerException:
> {noformat}
> select a, count(distinct 1), count_min_sketch(distinct b, 0.5d, 0.5d, 1)
> from values (1, 2), (4, 5) as data(a, b)
> group by a;
> {noformat}
> If you change the literal used in the first aggregation, then the query succeeds:
> {noformat}
> select a, count(distinct 2), count_min_sketch(distinct b, 0.5d, 0.5d, 1)
> from values (1, 2), (4, 5) as data(a, b)
> group by a;
> +---+-----------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
> |a |count(DISTINCT 2)|count_min_sketch(DISTINCT b, 0.5, 0.5, 1) |
> +---+-----------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
> |1 |1 |[00 00 00 01 00 00 00 00 00 00 00 01 00 00 00 01 00 00 00 04 00 00 00 00 5D 8D 6A B9 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00]|
> |4 |1 |[00 00 00 01 00 00 00 00 00 00 00 01 00 00 00 01 00 00 00 04 00 00 00 00 5D 8D 6A B9 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00]|
> +---+-----------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org