You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Ben Hurdelhey (Jira)" <ji...@apache.org> on 2025/08/18 12:20:00 UTC
[jira] [Created] (SPARK-53311) Fix MISSING_AGGREGATION for nondeterministic expression in both groupby and aggregate
Ben Hurdelhey created SPARK-53311:
-------------------------------------
Summary: Fix MISSING_AGGREGATION for nondeterministic expression in both groupby and aggregate
Key: SPARK-53311
URL: https://issues.apache.org/jira/browse/SPARK-53311
Project: Spark
Issue Type: Bug
Components: PySpark, Spark Core
Affects Versions: 4.1.0
Reporter: Ben Hurdelhey
Make PullOutNonDeterministic use canonicalized expressions to dedup group and aggregate expressions. This affects pyspark udfs in particular. Example:
```
from pyspark.sql.functions import col, avg, udf
pythonUDF = udf(lambda x: x).asNondeterministic()
spark.range(10)\
.selectExpr("id", "id % 3 as value")\
.groupBy(pythonUDF(col("value")))\
.agg(avg("id"), pythonUDF(col("value")))\
.explain(extended=True)
```
Currently results in a plan like this:
```
Aggregate [_nondeterministic#15], [_nondeterministic#15 AS dummyNondeterministicUDF(value)#12, avg(id#0L) AS avg(id)#13, dummyNondeterministicUDF(value#6L)#8 AS dummyNondeterministicUDF(value)#14]
+- Project [id#0L, value#6L, dummyNondeterministicUDF(value#6L)#7 AS _nondeterministic#15]
+- Project [id#0L, (id#0L % cast(3 as bigint)) AS value#6L]
+- Range (0, 10, step=1, splits=Some(2))```
and then throws:
```
[[MISSING_AGGREGATION|https://docs.databricks.com/error-messages/error-classes.html#missing_aggregation]] The non-aggregating expression "value" is based on columns which are not participating in the GROUP BY clause. Add the columns or the expression to the GROUP BY, aggregate the expression, or use "any_value(value)" if you do not care which of the values within a group is returned. SQLSTATE: 42803```
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org