You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/07/17 11:30:23 UTC
(datafusion) 01/01: feat: Support Struct type in approx_distinct (#23663)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-23663-fb8fe7ae61efc12c6c09a00d025e35c750e2ee86
in repository https://gitbox.apache.org/repos/asf/datafusion.git
commit 78be510e90821c748fcf4e3445d00cfa9165d860
Author: Michael Kleen <mk...@gmail.com>
AuthorDate: Fri Jul 17 13:29:58 2026 +0200
feat: Support Struct type in approx_distinct (#23663)
## Which issue does this PR close?
- Relates to https://github.com/apache/datafusion/issues/22989 but does
not close it. More types are coming.
## Rationale for this change
- Support the Arrow type `Struct` type for `approx_distinct`
## What changes are included in this PR?
- Enable `HLLAccumulator` and `HllGroupsAccumulator` to support `Struct`
- Tests for the non-grouped and grouped path as part of `aggregate.slt`
## Are these changes tested?
Yes and results are compared to DuckDB:
```sql
D CREATE TABLE approx_distinct_struct_test AS SELECT * FROM (VALUES
(1, {'a': 1, 'b': 2}), (1, {'a': 1, 'b': 2}), (1, {'a': 3, 'b': 3}),
(2, {'a': 4, 'b': 4}), (2, NULL),
(3, NULL), (3, NULL),
(4, {'a': 5, 'b': 5})
) AS t(g, s);
D SELECT approx_count_distinct(s) FROM approx_distinct_struct_test WHERE g = 1;
┌──────────────────────────┐
│ approx_count_distinct(s) │
│ int64 │
├──────────────────────────┤
│ 2 │
└──────────────────────────┘
D SELECT g, approx_count_distinct(s) FROM approx_distinct_struct_test GROUP BY g ORDER BY g;
┌───────┬──────────────────────────┐
│ g │ approx_count_distinct(s) │
│ int32 │ int64 │
├───────┼──────────────────────────┤
│ 1 │ 2 │
│ 2 │ 1 │
│ 3 │ 0 │
│ 4 │ 1 │
└───────┴──────────────────────────┘
D SELECT approx_count_distinct(s) FROM approx_distinct_struct_test;
┌──────────────────────────┐
│ approx_count_distinct(s) │
│ int64 │
├──────────────────────────┤
│ 4 │
└──────────────────────────┘
```
## Are there any user-facing changes?
Yes, `approx_distinct` supports now `Struct` but no breaking changes.
---
.../functions-aggregate/src/approx_distinct.rs | 2 ++
datafusion/sqllogictest/test_files/aggregate.slt | 38 ++++++++++++++++++++++
2 files changed, 40 insertions(+)
diff --git a/datafusion/functions-aggregate/src/approx_distinct.rs b/datafusion/functions-aggregate/src/approx_distinct.rs
index 19bb807ffa..6d3ea3a8dd 100644
--- a/datafusion/functions-aggregate/src/approx_distinct.rs
+++ b/datafusion/functions-aggregate/src/approx_distinct.rs
@@ -843,6 +843,7 @@ impl AggregateUDFImpl for ApproxDistinct {
| DataType::ListView(_)
| DataType::LargeListView(_)
| DataType::Map(_, _)
+ | DataType::Struct(_)
| DataType::LargeBinary => Box::new(HLLAccumulator::new()),
DataType::Null => {
Box::new(NoopAccumulator::new(ScalarValue::UInt64(Some(0))))
@@ -920,6 +921,7 @@ fn is_hll_groups_type(data_type: &DataType) -> bool {
| DataType::ListView(_)
| DataType::LargeListView(_)
| DataType::Map(_, _)
+ | DataType::Struct(_)
)
}
diff --git a/datafusion/sqllogictest/test_files/aggregate.slt b/datafusion/sqllogictest/test_files/aggregate.slt
index fa12f055a6..cbb9c5d031 100644
--- a/datafusion/sqllogictest/test_files/aggregate.slt
+++ b/datafusion/sqllogictest/test_files/aggregate.slt
@@ -2133,6 +2133,44 @@ SELECT approx_distinct(m) FROM approx_distinct_map_test;
statement ok
DROP TABLE approx_distinct_map_test;
+# Struct
+statement ok
+CREATE TABLE approx_distinct_struct_test AS SELECT * FROM (VALUES
+ (1, named_struct('a', 1, 'b', 2)), (1, named_struct('a', 1, 'b', 2)), (1, named_struct('a', 3, 'b', 3)),
+ (2, named_struct('a', 4, 'b', 4)), (2, NULL),
+ (3, NULL), (3, NULL),
+ (4, named_struct('a', 5, 'b', 5))
+) AS t(g, s);
+
+# Struct non-grouped
+query I
+SELECT approx_distinct(s) FROM approx_distinct_struct_test WHERE g = 1;
+----
+2
+
+# Struct grouped
+# Group 1 -> {{a:1,b:2},{a:3,b:3}}=2,
+# Group 2 -> {{a:4,b:4}}=1 (NULL excluded),
+# Group 3 -> all null=0,
+# Group 4 -> {{a:5,b:5}}=1
+query II
+SELECT g, approx_distinct(s) FROM approx_distinct_struct_test GROUP BY g ORDER BY g;
+----
+1 2
+2 1
+3 0
+4 1
+
+# The non-group path must agree with the grouped path on
+# the same data, and distinct structs across groups are still counted overall.
+query I
+SELECT approx_distinct(s) FROM approx_distinct_struct_test;
+----
+4
+
+statement ok
+DROP TABLE approx_distinct_struct_test;
+
# Integers (Int32): group 1 -> {10,20}=2, group 2 -> {30,40}=2, group 3 -> 0, group 4 -> {50}=1
query II
SELECT g, approx_distinct(i) FROM approx_distinct_group_test GROUP BY g ORDER BY g;
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org