You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (Jira)" <ji...@apache.org> on 2026/02/15 19:18:00 UTC
[jira] [Assigned] (SPARK-55533) Support IGNORE NULLS / RESPECT NULLS for collect_set
[ https://issues.apache.org/jira/browse/SPARK-55533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun reassigned SPARK-55533:
-------------------------------------
Assignee: Kent Yao
> Support IGNORE NULLS / RESPECT NULLS for collect_set
> ----------------------------------------------------
>
> Key: SPARK-55533
> URL: https://issues.apache.org/jira/browse/SPARK-55533
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Kent Yao
> Assignee: Kent Yao
> Priority: Major
> Labels: pull-request-available
>
> Currently, collect_list/array_agg supports IGNORE NULLS and RESPECT NULLS syntax (added in SPARK-55256), but collect_set does not.
> collect_set always ignores null values (Hive semantics). This ticket adds explicit IGNORE NULLS / RESPECT NULLS support to collect_set, mirroring the existing collect_list behavior:
> - collect_set(expr) — default, skips nulls (unchanged behavior)
> - collect_set(expr) IGNORE NULLS — explicitly skips nulls (same as default)
> - collect_set(expr) RESPECT NULLS — includes null in the result set if present
> Implementation approach:
> - Add ignoreNulls parameter to CollectSet (mirrors CollectList pattern)
> - Wire CollectSet into FunctionResolution.applyIgnoreNulls
> - Handle null safely in eval() for BinaryType
> Note: The SQL standard defines no SET_AGG function. collect_set is Spark-proprietary; the closest standard equivalent is ARRAY_AGG(DISTINCT ...). IGNORE NULLS / RESPECT NULLS is not defined for aggregate functions in the standard (only for window functions), so this is a Spark extension for consistency across collection aggregates.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org