You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Ben Hurdelhey (Jira)" <ji...@apache.org> on 2025/08/18 12:20:00 UTC

[jira] [Created] (SPARK-53311) Fix MISSING_AGGREGATION for nondeterministic expression in both groupby and aggregate

Ben Hurdelhey created SPARK-53311:
-------------------------------------

             Summary: Fix MISSING_AGGREGATION for nondeterministic expression in both groupby and aggregate
                 Key: SPARK-53311
                 URL: https://issues.apache.org/jira/browse/SPARK-53311
             Project: Spark
          Issue Type: Bug
          Components: PySpark, Spark Core
    Affects Versions: 4.1.0
            Reporter: Ben Hurdelhey


Make PullOutNonDeterministic use canonicalized expressions to dedup group and  aggregate expressions. This affects pyspark udfs in particular. Example:

 
```
from pyspark.sql.functions import col, avg, udf

pythonUDF = udf(lambda x: x).asNondeterministic()

spark.range(10)\
.selectExpr("id", "id % 3 as value")\
.groupBy(pythonUDF(col("value")))\
.agg(avg("id"), pythonUDF(col("value")))\
.explain(extended=True)
```
 
Currently results in a plan like this:
 
```
Aggregate [_nondeterministic#15], [_nondeterministic#15 AS dummyNondeterministicUDF(value)#12, avg(id#0L) AS avg(id)#13, dummyNondeterministicUDF(value#6L)#8 AS dummyNondeterministicUDF(value)#14]
+- Project [id#0L, value#6L, dummyNondeterministicUDF(value#6L)#7 AS _nondeterministic#15]
   +- Project [id#0L, (id#0L % cast(3 as bigint)) AS value#6L]
      +- Range (0, 10, step=1, splits=Some(2))```
 
and then throws:
```
[[MISSING_AGGREGATION|https://docs.databricks.com/error-messages/error-classes.html#missing_aggregation]] The non-aggregating expression "value" is based on columns which are not participating in the GROUP BY clause. Add the columns or the expression to the GROUP BY, aggregate the expression, or use "any_value(value)" if you do not care which of the values within a group is returned. SQLSTATE: 42803```



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org