You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "mkleen (via GitHub)" <gi...@apache.org> on 2026/07/20 08:58:55 UTC
[PR] feat: Support Union type in approx_distinct [datafusion]
mkleen opened a new pull request, #23714:
URL: https://github.com/apache/datafusion/pull/23714
## Which issue does this PR close?
- Relates to https://github.com/apache/datafusion/issues/22989 but does not close it. More types are coming.
## Rationale for this change
- Support the Arrow type `Union` type for `approx_distinct`
## What changes are included in this PR?
- Enable `HLLAccumulator` and `HllGroupsAccumulator` to support `Union`
- Tests for the non-grouped and grouped path as part of `approx_distinct.rs`
## Are these changes tested?
Yes, no slt because `Union` has no mapped SQL Type.
## Are there any user-facing changes?
Yes, `approx_distinct` supports now `Union` but no breaking changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Support Union type in approx_distinct [datafusion]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on PR #23714:
URL: https://github.com/apache/datafusion/pull/23714#issuecomment-5056250352
@Jefffrey thanks for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Support Union type in approx_distinct [datafusion]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on code in PR #23714:
URL: https://github.com/apache/datafusion/pull/23714#discussion_r3630571566
##########
datafusion/sqllogictest/src/test_context.rs:
##########
@@ -593,6 +597,43 @@ fn register_union_table(ctx: &SessionContext) {
ctx.register_batch("union_table", batch).unwrap();
}
+fn register_approx_distinct_union_table(ctx: &SessionContext) {
Review Comment:
I added a separate batch which fits better for testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Support Union type in approx_distinct [datafusion]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #23714:
URL: https://github.com/apache/datafusion/pull/23714#discussion_r3614062955
##########
datafusion/functions-aggregate/src/approx_distinct.rs:
##########
@@ -1078,6 +1080,66 @@ mod tests {
assert_count_numerical_acc_and_group_acc::<Decimal256Type>(decimal_256, 6);
}
+ #[test]
+ fn union_support_hll_acc_and_group_acc() {
Review Comment:
not possible to do via SLT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Support Union type in approx_distinct [datafusion]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey merged PR #23714:
URL: https://github.com/apache/datafusion/pull/23714
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Support Union type in approx_distinct [datafusion]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on code in PR #23714:
URL: https://github.com/apache/datafusion/pull/23714#discussion_r3614948860
##########
datafusion/functions-aggregate/src/approx_distinct.rs:
##########
@@ -1078,6 +1080,66 @@ mod tests {
assert_count_numerical_acc_and_group_acc::<Decimal256Type>(decimal_256, 6);
}
+ #[test]
+ fn union_support_hll_acc_and_group_acc() {
Review Comment:
It's tricky via SLT. There is no sql union type but only union operations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Support Union type in approx_distinct [datafusion]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on code in PR #23714:
URL: https://github.com/apache/datafusion/pull/23714#discussion_r3614948860
##########
datafusion/functions-aggregate/src/approx_distinct.rs:
##########
@@ -1078,6 +1080,66 @@ mod tests {
assert_count_numerical_acc_and_group_acc::<Decimal256Type>(decimal_256, 6);
}
+ #[test]
+ fn union_support_hll_acc_and_group_acc() {
Review Comment:
It's tricky via SLT. There is no sql union type but only union operators in datafusion.
##########
datafusion/functions-aggregate/src/approx_distinct.rs:
##########
@@ -1078,6 +1080,66 @@ mod tests {
assert_count_numerical_acc_and_group_acc::<Decimal256Type>(decimal_256, 6);
}
+ #[test]
+ fn union_support_hll_acc_and_group_acc() {
Review Comment:
It's tricky via SLT. There is no SQL union type but only union operators in datafusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Support Union type in approx_distinct [datafusion]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #23714:
URL: https://github.com/apache/datafusion/pull/23714#discussion_r3619284855
##########
datafusion/functions-aggregate/src/approx_distinct.rs:
##########
@@ -1078,6 +1080,66 @@ mod tests {
assert_count_numerical_acc_and_group_acc::<Decimal256Type>(decimal_256, 6);
}
+ #[test]
+ fn union_support_hll_acc_and_group_acc() {
Review Comment:
what if we add more rows here:
https://github.com/apache/datafusion/blob/3a2bc0a0a5673386ab2238b545dbedb82ce6b92b/datafusion/sqllogictest/src/test_context.rs#L560-L594
then can use in slts like here
https://github.com/apache/datafusion/blob/3a2bc0a0a5673386ab2238b545dbedb82ce6b92b/datafusion/sqllogictest/test_files/union_function.slt#L18-L30
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Support Union type in approx_distinct [datafusion]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #23714:
URL: https://github.com/apache/datafusion/pull/23714#issuecomment-5056140223
thanks @mkleen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Support Union type in approx_distinct [datafusion]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on code in PR #23714:
URL: https://github.com/apache/datafusion/pull/23714#discussion_r3614948860
##########
datafusion/functions-aggregate/src/approx_distinct.rs:
##########
@@ -1078,6 +1080,66 @@ mod tests {
assert_count_numerical_acc_and_group_acc::<Decimal256Type>(decimal_256, 6);
}
+ #[test]
+ fn union_support_hll_acc_and_group_acc() {
Review Comment:
It's tricky via SLT. There is no SQL union type but only union operations in datafusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Support Union type in approx_distinct [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #23714:
URL: https://github.com/apache/datafusion/pull/23714#issuecomment-5020978211
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/23714?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 80.69%. Comparing base ([`3b1ce16`](https://app.codecov.io/gh/apache/datafusion/commit/3b1ce16be6beb3377c59ec36fd9f8ae62998c4c0?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`9096c52`](https://app.codecov.io/gh/apache/datafusion/commit/9096c52e77e322b951e73911774715988b4fc7c0?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #23714 +/- ##
=======================================
Coverage 80.69% 80.69%
=======================================
Files 1088 1088
Lines 367766 367813 +47
Branches 367766 367813 +47
=======================================
+ Hits 296781 296820 +39
- Misses 53276 53279 +3
- Partials 17709 17714 +5
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/23714?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Support Union type in approx_distinct [datafusion]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on code in PR #23714:
URL: https://github.com/apache/datafusion/pull/23714#discussion_r3629763565
##########
datafusion/functions-aggregate/src/approx_distinct.rs:
##########
@@ -1078,6 +1080,66 @@ mod tests {
assert_count_numerical_acc_and_group_acc::<Decimal256Type>(decimal_256, 6);
}
+ #[test]
+ fn union_support_hll_acc_and_group_acc() {
Review Comment:
Good idea!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org