You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Rich-T-kid (via GitHub)" <gi...@apache.org> on 2026/08/05 18:07:04 UTC
[I] Improve NDV statistics for Parquet columns [datafusion]
Rich-T-kid opened a new issue, #24114:
URL: https://github.com/apache/datafusion/issues/24114
### Is your feature request related to a problem or challenge?
Deciding whether to dictionary-encode a column requires knowing its cardinality (NDV), Datafusion does support https://docs.rs/datafusion/latest/datafusion/common/struct.ColumnStatistics.html but we need to validate this.
### Describe the solution you'd like
Improve NDV statistics collection focused on Parquet, and get these stats into DataFusion in a form the optimizer can actually use (e.g. exposed on `ColumnStatistics`).
### Describe alternatives you've considered
_No response_
### Additional context
Direct sub-issue of #22891
piece of Epic : #24111
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Improve NDV statistics for Parquet columns (parquet RLE & Dictionary array focus) [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 closed issue #24114: Improve NDV statistics for Parquet columns (parquet RLE & Dictionary array focus)
URL: https://github.com/apache/datafusion/issues/24114
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Improve NDV statistics for Parquet columns (parquet RLE & Dictionary array focus) [datafusion]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on issue #24114:
URL: https://github.com/apache/datafusion/issues/24114#issuecomment-5350977451
with https://github.com/apache/arrow-rs/issues/10650 being closed in arrow-rs we just need to wait for datafusion to bump the version so we can use the new methods
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Improve NDV statistics for Parquet columns (parquet RLE & Dictionary array focus) [datafusion]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on issue #24114:
URL: https://github.com/apache/datafusion/issues/24114#issuecomment-5752995684
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Improve NDV statistics for Parquet columns (parquet RLE & Dictionary array focus) [datafusion]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on issue #24114:
URL: https://github.com/apache/datafusion/issues/24114#issuecomment-5673142326
waiting for arrow-rs v60.0.0 fyi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org