You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "2010YOUY01 (via GitHub)" <gi...@apache.org> on 2026/06/17 02:04:27 UTC
[I] Support more types for `approx_distinct` function [datafusion]
2010YOUY01 opened a new issue, #22989:
URL: https://github.com/apache/datafusion/issues/22989
### Is your feature request related to a problem or challenge?
#22921 has refactored `approx_distinct` into a more generic implementation over different input types, now it's easy to support additional arg types. (Thanks to @Jefffrey for pointing it out in the review)
### Describe the solution you'd like
1. Identify the unsupported types
2. Plan them into the generic implementation of `Accumulator` and `GroupsAccumulator` (see #22921 for details)
3. Add `sqllogictest`s
Note we don't have to support all applicable types at once, some complex type might get tricky to test, so I prefer to do this incrementally
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support more types for `approx_distinct` function [datafusion]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on issue #22989:
URL: https://github.com/apache/datafusion/issues/22989#issuecomment-5013795331
the way i see it, if regular distinct supports the data type, approx distinct should also support it; given it seems regular distinct supports all types we should follow suit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support more types for `approx_distinct` function [datafusion]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on issue #22989:
URL: https://github.com/apache/datafusion/issues/22989#issuecomment-4787791956
I had a look at DuckDB. DuckDB doesn't declare the supported types in their docs for approx_count_distinct https://duckdb.org/docs/lts/sql/functions/aggregates.
I did some manual testing and it seems it supports every each data type:
| Category | Types |
|---|---|
| Signed int | `TINYINT`, `SMALLINT`, `INTEGER`, `BIGINT`, `HUGEINT` |
| Unsigned int | `UTINYINT`, `USMALLINT`, `UINTEGER`, `UBIGINT` |
| Float | `FLOAT`, `DOUBLE`, `DECIMAL` |
| String/binary | `VARCHAR`, `BLOB`, `BIT`, `UUID` |
| Temporal | `DATE`, `TIME`, `TIMESTAMP`, `TIMESTAMPTZ`, `INTERVAL` |
| Other | `BOOLEAN`, `JSON` |
| Nested | `LIST`, `STRUCT`, `MAP` |
Maybe let's focus as a start on the datatypes which a straight-forward to support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support more types for `approx_distinct` function [datafusion]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on issue #22989:
URL: https://github.com/apache/datafusion/issues/22989#issuecomment-4726323868
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support more types for `approx_distinct` function [datafusion]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on issue #22989:
URL: https://github.com/apache/datafusion/issues/22989#issuecomment-5015075351
Ok, thanks for the feedback!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support more types for `approx_distinct` function [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #22989:
URL: https://github.com/apache/datafusion/issues/22989#issuecomment-4782072934
FWIW some types, such as floats, likely don't make sense to implement DISTINCT for (as we found in https://github.com/apache/datafusion/pull/23084).
It would probably be good to review what other database implementations do before implementing it for datafusion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support more types for `approx_distinct` function [datafusion]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on issue #22989:
URL: https://github.com/apache/datafusion/issues/22989#issuecomment-5012760509
We won't support `Float` types. What about `RunEndEncoded`, `Union` and `Dictionary` ? They are no SQL types mapped to them. Does it make sense to support them?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support more types for `approx_distinct` function [datafusion]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on issue #22989:
URL: https://github.com/apache/datafusion/issues/22989#issuecomment-4788206350
The arrow-types which in datafusion are:
| Category | Arrow Type |
|---|---|
| Float | `Float16` |
| Float | `Float32` |
| Float | `Float64` |
| Decimal | `Decimal32` |
| Decimal | `Decimal64` |
| Decimal | `Decimal128` |
| Decimal | `Decimal256` |
| Temporal | `Duration` |
| Temporal | `Interval` |
| Binary | `BinaryView` |
| Binary | `FixedSizeBinary` |
| List/Nested | `List` |
| List/Nested | `LargeList` |
| List/Nested | `ListView` |
| List/Nested | `LargeListView` |
| List/Nested | `FixedSizeList` |
| Complex | `Struct` |
| Complex | `Map` |
| Complex | `Union` |
| Complex | `Dictionary` |
| Complex | `RunEndEncoded` |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support more types for `approx_distinct` function [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 closed issue #22989: Support more types for `approx_distinct` function
URL: https://github.com/apache/datafusion/issues/22989
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org