You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "xudong963 (via GitHub)" <gi...@apache.org> on 2025/03/18 07:57:23 UTC
[I] Support `merge` for `Distribution` [datafusion]
xudong963 opened a new issue, #15290:
URL: https://github.com/apache/datafusion/issues/15290
### Is your feature request related to a problem or challenge?
I'm working on the ticket: https://github.com/apache/datafusion/issues/10316.
Given that, we'll replace all `Precision` with `Distribution`: https://github.com/synnada-ai/datafusion-upstream/pull/63. So, while I make the design for #10316, I presumably use `Distribution` in statistics.
There is a spot where I'll do the `merge` for statistics, and it'll be spread to the `Distribution`.
The specific case is that I need to compute the partition-level statistics, aka, files will be grouped as the filegroup, each file group will be treated as a partition, and different partitions will be processed in parallel. So, the partition-level statistics will be from the merge of the files in a filegroup.
### Describe the solution you'd like
Create a function that combines their statistical properties into a new distribution. The most appropriate approach is to create a GenericDistribution that approximates the mixture of the two input distributions.
```rust
pub fn merge_distributions(a: &Distribution, b: &Distribution) -> Result<Distribution> {
...
}
```
---
I'll open a PR and we can do more discussions based on the PR.
### Describe alternatives you've considered
No
### Additional context
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support `merge` for `Distribution` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #15290:
URL: https://github.com/apache/datafusion/issues/15290#issuecomment-2732107381
> I'm working on the ticket: https://github.com/apache/datafusion/issues/10316.
> Create a function that combines their statistical properties into a new distribution. The most appropriate approach is to create a GenericDistribution that approximates the mixture of the two input distributions.
I think implementing the actual analysis in
- https://github.com/apache/datafusion/issues/10316.
Will require an accurate distribution (not just an approximation). Using ProgressiveEval requires (for correctness) that we know the ranges do not overlap
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support `merge` for `Distribution` [datafusion]
Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 commented on issue #15290:
URL: https://github.com/apache/datafusion/issues/15290#issuecomment-2732132955
> Will require an accurate distribution (not just an approximation
Yes, it depends on whether each distribution is accurate, if they're, the merged distribution should be accurate, or we should merge them conservatively
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support `merge` for `Distribution` [datafusion]
Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 commented on issue #15290:
URL: https://github.com/apache/datafusion/issues/15290#issuecomment-2732391652
There is a proposal: https://github.com/apache/datafusion/pull/15296
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support `merge` for `Distribution` [datafusion]
Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 closed issue #15290: Support `merge` for `Distribution`
URL: https://github.com/apache/datafusion/issues/15290
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org