You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Rich-T-kid (via GitHub)" <gi...@apache.org> on 2026/08/05 18:04:13 UTC
[I] Allow users to define the cardinality threshold for dictionary reads [datafusion]
Rich-T-kid opened a new issue, #24113:
URL: https://github.com/apache/datafusion/issues/24113
### Is your feature request related to a problem or challenge?
_No response_
### Describe the solution you'd like
Add a config option to set a cardinality threshold, so columns above it are read as regular strings instead of dictionaries.
for example
```
SET datafusion.parquet.dict_threshold = 0.2 # dataset must have less than 20% unqiue values
```
or
```
SET datafusion.parquet.dict_threshold = 200 # 200 unique values max
```
we can support both but im leaning towards supporting a percentage threshold.
### Describe alternatives you've considered
_No response_
### Additional context
#24111
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Allow users to define the cardinality threshold for dictionary reads [datafusion]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on issue #24113:
URL: https://github.com/apache/datafusion/issues/24113#issuecomment-5195683018
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Allow users to define the cardinality threshold for dictionary reads [datafusion]
Posted by "Rich-T-kid (via GitHub) via github" <gi...@datafusion.apache.org>.
Rich-T-kid commented on issue #24113:
URL: https://github.com/apache/datafusion/issues/24113#issuecomment-6834271535
After extensive investigation and benchmarking, I think we should close this without adding a standalone NDV threshold.
NDV by itself is not a reliable predictor of whether dictionary reads will help. Even at medium or high cardinality, dictionaries can be beneficial when data pages remain dictionary encoded. Conversely, when a column chunk mixes `RLE_DICTIONARY` and plain pages, the Arrow reader may need to expand and rebuild the dictionary during encoding transitions, which can erase the benefit or cause regressions. More context on that reader behavior is tracked in apache/arrow-rs#12195.
I think the narrower encoding-aware direction is worth pursuing, but it belongs as part of the broader optimization work in #24117 rather than as a user-configured NDV threshold. Closing this specific issue in favor of that direction.
cc @asolimando
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Allow users to define the cardinality threshold for dictionary reads [datafusion]
Posted by "Rich-T-kid (via GitHub) via github" <gi...@datafusion.apache.org>.
Rich-T-kid closed issue #24113: Allow users to define the cardinality threshold for dictionary reads
URL: https://github.com/apache/datafusion/issues/24113
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Allow users to define the cardinality threshold for dictionary reads [datafusion]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on issue #24113:
URL: https://github.com/apache/datafusion/issues/24113#issuecomment-5274351619
important in this context : https://github.com/apache/arrow-rs/issues/8608#issuecomment-5262298172
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Allow users to define the cardinality threshold for dictionary reads [datafusion]
Posted by "Rich-T-kid (via GitHub) via github" <gi...@datafusion.apache.org>.
Rich-T-kid commented on issue #24113:
URL: https://github.com/apache/datafusion/issues/24113#issuecomment-6938489270
Reopening this because my earlier conclusion was too broad.
The mixed `RLE_DICTIONARY`/plain-page failure mode is real, but it can be guarded independently from NDV. The new `dictionary_read_max_distinct_values` option fails closed unless every row group has NDV statistics and page-encoding statistics prove that every data page is dictionary encoded. It then compares the configured cap against the sum of row-group NDVs, a conservative per-file upper bound, and rechecks each file when it is opened.
That keeps the user-controlled threshold requested here while excluding the encoding transition that caused the earlier regressions. I have also rerun the full TPC-H and TPC-DS SF1 matrices plus focused confirmation runs; no regression above 5% reproduced in both independent pairs. The PR will include the full benchmark method and results.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org