You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Dandandan (via GitHub)" <gi...@apache.org> on 2026/04/17 08:38:50 UTC

[I] Initialize TopK from file / rowgroup / .. statistics [datafusion]

Dandandan opened a new issue, #21691:
URL: https://github.com/apache/datafusion/issues/21691

   Based on this PR and morsel improvements, I am also thinking we could initialize the TopK statistics from column stats (at least for single columns) and make the initial threshold much tighter based on min/max statistics (at file / rowgroup/page level):
   
   * We have a file/rowgroup with more than K (from TopK) amount of rows
   * We have a single order by column (directly after scan)
   * We can initialize/update the TopK using max (or min) statistics
   * Also, if the new bound is smaller / bigger than the current TopK, we could update it to the tighter bound
   
   This I think might help making initial threshold much tighter instead of having to read all the first row groups using not-initialized TopK.
   
   _Originally posted by @Dandandan in https://github.com/apache/datafusion/issues/21580#issuecomment-4266553697_
               


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org