You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/07/14 10:07:36 UTC
(datafusion) branch gh-readonly-queue/main/pr-23526-95cda37241468514a810301a50fb922d459ff4bd created (now 84eaef9b0f)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-23526-95cda37241468514a810301a50fb922d459ff4bd
in repository https://gitbox.apache.org/repos/asf/datafusion.git
at 84eaef9b0f feat: Support Map type in approx_distinct (#23526)
This branch includes the following new commits:
new 84eaef9b0f feat: Support Map type in approx_distinct (#23526)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion) 01/01: feat: Support Map type in approx_distinct (#23526)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-23526-95cda37241468514a810301a50fb922d459ff4bd
in repository https://gitbox.apache.org/repos/asf/datafusion.git
commit 84eaef9b0fad83f61c84ed854bf2be43f1089ecf
Author: Michael Kleen <mk...@gmail.com>
AuthorDate: Tue Jul 14 11:53:04 2026 +0200
feat: Support Map type in approx_distinct (#23526)
## Which issue does this PR close?
- Relates to https://github.com/apache/datafusion/issues/22989 but does
not close it. More types are coming.
## Rationale for this change
- Support the Arrow types `Map` type for `approx_distinct`
## What changes are included in this PR?
- Enable `HLLAccumulator` and `HllGroupsAccumulator` to support `Map`
- Tests for the non-grouped and grouped path as part of `aggregate.slt`
## Are these changes tested?
Yes and results are compared to DuckDB:
```
D CREATE TABLE approx_distinct_map_test AS SELECT * FROM (VALUES
(1, MAP {'a': 1, 'b': 2}), (1, MAP {'a': 1, 'b': 2}), (1, MAP {'c': 3}),
(2, MAP {'d': 4}), (2, NULL),
(3, NULL), (3, NULL),
(4, MAP {'e': 5})
) AS t(g, m);
D SELECT approx_count_distinct(m) FROM approx_distinct_map_test WHERE g = 1;
┌──────────────────────────┐
│ approx_count_distinct(m) │
│ int64 │
├──────────────────────────┤
│ 2 │
└──────────────────────────┘
D SELECT g, approx_count_distinct(m) FROM approx_distinct_map_test GROUP BY g ORDER BY g;
┌───────┬──────────────────────────┐
│ g │ approx_count_distinct(m) │
│ int32 │ int64 │
├───────┼──────────────────────────┤
│ 1 │ 2 │
│ 2 │ 1 │
│ 3 │ 0 │
│ 4 │ 1 │
└───────┴──────────────────────────┘
D SELECT approx_count_distinct(m) FROM approx_distinct_map_test;
┌──────────────────────────┐
│ approx_count_distinct(m) │
│ int64 │
├──────────────────────────┤
│ 4 │
└──────────────────────────┘
```
## Are there any user-facing changes?
Yes, `approx_distinct` supports now `Map` but no breaking changes.
Co-authored-by: Nuno Faria <nu...@gmail.com>
---
.../functions-aggregate/src/approx_distinct.rs | 2 ++
datafusion/sqllogictest/test_files/aggregate.slt | 38 ++++++++++++++++++++++
2 files changed, 40 insertions(+)
diff --git a/datafusion/functions-aggregate/src/approx_distinct.rs b/datafusion/functions-aggregate/src/approx_distinct.rs
index 15df82a79a..19bb807ffa 100644
--- a/datafusion/functions-aggregate/src/approx_distinct.rs
+++ b/datafusion/functions-aggregate/src/approx_distinct.rs
@@ -842,6 +842,7 @@ impl AggregateUDFImpl for ApproxDistinct {
| DataType::FixedSizeList(_, _)
| DataType::ListView(_)
| DataType::LargeListView(_)
+ | DataType::Map(_, _)
| DataType::LargeBinary => Box::new(HLLAccumulator::new()),
DataType::Null => {
Box::new(NoopAccumulator::new(ScalarValue::UInt64(Some(0))))
@@ -918,6 +919,7 @@ fn is_hll_groups_type(data_type: &DataType) -> bool {
| DataType::FixedSizeList(_, _)
| DataType::ListView(_)
| DataType::LargeListView(_)
+ | DataType::Map(_, _)
)
}
diff --git a/datafusion/sqllogictest/test_files/aggregate.slt b/datafusion/sqllogictest/test_files/aggregate.slt
index e07c9e1b73..fa12f055a6 100644
--- a/datafusion/sqllogictest/test_files/aggregate.slt
+++ b/datafusion/sqllogictest/test_files/aggregate.slt
@@ -2095,6 +2095,44 @@ SELECT approx_distinct(arrow_cast(l, 'FixedSizeList(2, Int32)')) FROM approx_dis
statement ok
DROP TABLE approx_distinct_list_test;
+# Map
+statement ok
+CREATE TABLE approx_distinct_map_test AS SELECT * FROM (VALUES
+ (1, MAP {'a': 1, 'b': 2}), (1, MAP {'a': 1, 'b': 2}), (1, MAP {'c': 3}),
+ (2, MAP {'d': 4}), (2, NULL),
+ (3, NULL), (3, NULL),
+ (4, MAP {'e': 5})
+) AS t(g, m);
+
+# Map non-grouped
+query I
+SELECT approx_distinct(m) FROM approx_distinct_map_test WHERE g = 1;
+----
+2
+
+# Map grouped
+# Group 1 -> {{a:1,b:2},{c:3}}=2,
+# Group 2 -> {{d:4}}=1 (NULL excluded),
+# Group 3 -> all null=0,
+# Group 4 -> {{e:5}}=1
+query II
+SELECT g, approx_distinct(m) FROM approx_distinct_map_test GROUP BY g ORDER BY g;
+----
+1 2
+2 1
+3 0
+4 1
+
+# The non-group path must agree with the grouped path on
+# the same data, and distinct maps across groups are still counted overall.
+query I
+SELECT approx_distinct(m) FROM approx_distinct_map_test;
+----
+4
+
+statement ok
+DROP TABLE approx_distinct_map_test;
+
# Integers (Int32): group 1 -> {10,20}=2, group 2 -> {30,40}=2, group 3 -> 0, group 4 -> {50}=1
query II
SELECT g, approx_distinct(i) FROM approx_distinct_group_test GROUP BY g ORDER BY g;
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org