You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/09/16 13:41:26 UTC
(datafusion) branch asf-site updated: Publish built docs triggered by ddad4a821aee49586fd213ab9a4a77bb2826a076
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion.git
The following commit(s) were added to refs/heads/asf-site by this push:
new b1065831f0 Publish built docs triggered by ddad4a821aee49586fd213ab9a4a77bb2826a076
b1065831f0 is described below
commit b1065831f01f8a95cd62a54e22b645afb2c1cdfc
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Wed Sep 16 13:41:20 2026 +0000
Publish built docs triggered by ddad4a821aee49586fd213ab9a4a77bb2826a076
---
.../library-user-guide/functions/adding-udfs.md.txt | 19 +++++++++++++++++++
library-user-guide/functions/adding-udfs.html | 19 +++++++++++++++++++
searchindex.js | 2 +-
3 files changed, 39 insertions(+), 1 deletion(-)
diff --git a/_sources/library-user-guide/functions/adding-udfs.md.txt b/_sources/library-user-guide/functions/adding-udfs.md.txt
index c3a40557a0..78e90dfa6b 100644
--- a/_sources/library-user-guide/functions/adding-udfs.md.txt
+++ b/_sources/library-user-guide/functions/adding-udfs.md.txt
@@ -1116,6 +1116,24 @@ impl Accumulator for GeometricMean {
}
```
+### Declaring how an Aggregate UDF treats `DISTINCT`
+
+By default DataFusion assumes an aggregate is sensitive to the `DISTINCT` modifier, which means the accumulator is expected to
+read `AccumulatorArgs::is_distinct` and deduplicate its input. Override
+[`AggregateUDFImpl::distinct_handling`] when that is not what your function does:
+
+- Return `DistinctHandling::Insensitive` when duplicates cannot change the result, that is, when merging a value the
+ accumulator has already seen is a no-op. `min`, `max`, `bool_and` and `bit_or` are all in this group. The optimizer
+ then plans `f(DISTINCT x)` as `f(x)`, which skips both the per-group hash set and the extra grouping stage that
+ `SingleDistinctToGroupBy` would otherwise introduce.
+- Return `DistinctHandling::Unsupported` when the accumulator does not implement `DISTINCT`: it does not read
+ `is_distinct`, or it rejects `DISTINCT` with an error. The planner must then deduplicate the input first or reject
+ the query. Today this is a declaration only; rejecting such queries at planning time is a follow-up change.
+- Leave the default `DistinctHandling::Sensitive` when the accumulator reads `AccumulatorArgs::is_distinct` and
+ deduplicates its input itself.
+
+Getting this wrong changes query results, so only claim `Insensitive` if your merge is genuinely idempotent.
+
### Registering an Aggregate UDF
To register a Aggregate UDF, you need to wrap the function implementation in a [`AggregateUDF`] struct and then register
@@ -1370,6 +1388,7 @@ async fn main() -> Result<()> {
[`aggregateudf`]: https://docs.rs/datafusion/latest/datafusion/logical_expr/struct.AggregateUDF.html
[`create_udaf`]: https://docs.rs/datafusion/latest/datafusion/logical_expr/fn.create_udaf.html
+[`aggregateudfimpl::distinct_handling`]: https://docs.rs/datafusion/latest/datafusion/logical_expr/trait.AggregateUDFImpl.html#method.distinct_handling
[`advanced_udaf.rs`]: https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/udf/advanced_udaf.rs
## Adding a Table UDF
diff --git a/library-user-guide/functions/adding-udfs.html b/library-user-guide/functions/adding-udfs.html
index 2df3be89ae..0a3fc34cfd 100644
--- a/library-user-guide/functions/adding-udfs.html
+++ b/library-user-guide/functions/adding-udfs.html
@@ -1136,6 +1136,24 @@ green,10.3,1996-04-12T12:05:04.000000000
<span class="p">}</span>
</pre></div>
</div>
+<section id="declaring-how-an-aggregate-udf-treats-distinct">
+<h3>Declaring how an Aggregate UDF treats <code class="docutils literal notranslate"><span class="pre">DISTINCT</span></code><a class="headerlink" href="#declaring-how-an-aggregate-udf-treats-distinct" title="Link to this heading">#</a></h3>
+<p>By default DataFusion assumes an aggregate is sensitive to the <code class="docutils literal notranslate"><span class="pre">DISTINCT</span></code> modifier, which means the accumulator is expected to
+read <code class="docutils literal notranslate"><span class="pre">AccumulatorArgs::is_distinct</span></code> and deduplicate its input. Override
+<a class="reference external" href="https://docs.rs/datafusion/latest/datafusion/logical_expr/trait.AggregateUDFImpl.html#method.distinct_handling"><code class="docutils literal notranslate"><span class="pre">AggregateUDFImpl::distinct_handling</span></code></a> when that is not what your function does:</p>
+<ul class="simple">
+<li><p>Return <code class="docutils literal notranslate"><span class="pre">DistinctHandling::Insensitive</span></code> when duplicates cannot change the result, that is, when merging a value the
+accumulator has already seen is a no-op. <code class="docutils literal notranslate"><span class="pre">min</span></code>, <code class="docutils literal notranslate"><span class="pre">max</span></code>, <code class="docutils literal notranslate"><span class="pre">bool_and</span></code> and <code class="docutils literal notranslate"><span class="pre">bit_or</span></code> are all in this group. The optimizer
+then plans <code class="docutils literal notranslate"><span class="pre">f(DISTINCT</span> <span class="pre">x)</span></code> as <code class="docutils literal notranslate"><span class="pre">f(x)</span></code>, which skips both the per-group hash set and the extra grouping stage that
+<code class="docutils literal notranslate"><span class="pre">SingleDistinctToGroupBy</span></code> would otherwise introduce.</p></li>
+<li><p>Return <code class="docutils literal notranslate"><span class="pre">DistinctHandling::Unsupported</span></code> when the accumulator does not implement <code class="docutils literal notranslate"><span class="pre">DISTINCT</span></code>: it does not read
+<code class="docutils literal notranslate"><span class="pre">is_distinct</span></code>, or it rejects <code class="docutils literal notranslate"><span class="pre">DISTINCT</span></code> with an error. The planner must then deduplicate the input first or reject
+the query. Today this is a declaration only; rejecting such queries at planning time is a follow-up change.</p></li>
+<li><p>Leave the default <code class="docutils literal notranslate"><span class="pre">DistinctHandling::Sensitive</span></code> when the accumulator reads <code class="docutils literal notranslate"><span class="pre">AccumulatorArgs::is_distinct</span></code> and
+deduplicates its input itself.</p></li>
+</ul>
+<p>Getting this wrong changes query results, so only claim <code class="docutils literal notranslate"><span class="pre">Insensitive</span></code> if your merge is genuinely idempotent.</p>
+</section>
<section id="registering-an-aggregate-udf">
<h3>Registering an Aggregate UDF<a class="headerlink" href="#registering-an-aggregate-udf" title="Link to this heading">#</a></h3>
<p>To register a Aggregate UDF, you need to wrap the function implementation in a <a class="reference external" href="https://docs.rs/datafusion/latest/datafusion/logical_expr/struct.AggregateUDF.html"><code class="docutils literal notranslate"><span class="pre">AggregateUDF</span></code></a> struct and then register
@@ -1475,6 +1493,7 @@ validation of the input <code class="docutils literal notranslate"><span class="
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#adding-an-aggregate-udf">Adding an Aggregate UDF</a><ul class="nav section-nav flex-column">
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#declaring-how-an-aggregate-udf-treats-distinct">Declaring how an Aggregate UDF treats <code class="docutils literal notranslate"><span class="pre">DISTINCT</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#registering-an-aggregate-udf">Registering an Aggregate UDF</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#returning-multiple-values-from-an-aggregate-udf">Returning multiple values from an Aggregate UDF</a></li>
</ul>
diff --git a/searchindex.js b/searchindex.js
index 17ab3f5a10..288d372273 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles":{"!=":[[77,"op-neq"]],"!~":[[77,"op-re-not-match"]],"!~*":[[77,"op-re-not-match-i"]],"!~~":[[77,"id19"]],"!~~*":[[77,"id20"]],"#":[[77,"op-bit-xor"]],"%":[[77,"op-modulo"]],"&":[[77,"op-bit-and"]],"(relation, name) tuples in logical fields and logical columns are unique":[[16,"relation-name-tuples-in-logical-fields-and-logical-columns-are-unique"]],"*":[[77,"op-multiply"]],"+":[[77,"op-plus"]],"-":[[77,"op-minus"]],"/":[[77,"op-divide"]],"1. Array Literal Con [...]
\ No newline at end of file
+Search.setIndex({"alltitles":{"!=":[[77,"op-neq"]],"!~":[[77,"op-re-not-match"]],"!~*":[[77,"op-re-not-match-i"]],"!~~":[[77,"id19"]],"!~~*":[[77,"id20"]],"#":[[77,"op-bit-xor"]],"%":[[77,"op-modulo"]],"&":[[77,"op-bit-and"]],"(relation, name) tuples in logical fields and logical columns are unique":[[16,"relation-name-tuples-in-logical-fields-and-logical-columns-are-unique"]],"*":[[77,"op-multiply"]],"+":[[77,"op-plus"]],"-":[[77,"op-minus"]],"/":[[77,"op-divide"]],"1. Array Literal Con [...]
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org