You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "milenkovicm (via GitHub)" <gi...@apache.org> on 2026/06/07 10:26:05 UTC
[I] Improve shuffle (column) statistics [datafusion-ballista]
milenkovicm opened a new issue, #1827:
URL: https://github.com/apache/datafusion-ballista/issues/1827
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Currently, shuffle collects basic statistics like the number of rows and bytes. This data is used downstream for some optimisation. However, I understand those statistics aren’t sufficient for join estimations, leading to incorrect decisions in some cases. No column based statistics such as null count or distinct count is collected during exchange.
**Describe the solution you'd like**
investigate what would be appropriate approach to collect null and distinct count for columns used in repartition expressions. Collecting null count may not be big problem but distinct count might and approximation may be used for it
**Describe alternatives you've considered**
**Additional context**
- collected statistic should be propagated to planners
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Improve shuffle (column) statistics [datafusion-ballista]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on issue #1827:
URL: https://github.com/apache/datafusion-ballista/issues/1827#issuecomment-4727491606
looking
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Improve shuffle (column) statistics [datafusion-ballista]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender closed issue #1827: Improve shuffle (column) statistics
URL: https://github.com/apache/datafusion-ballista/issues/1827
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org