You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "niebayes (via GitHub)" <gi...@apache.org> on 2025/11/14 02:32:25 UTC

[I] Question: should set FileScanConfig's batch_size to min(batch_size, limit) [datafusion]

niebayes opened a new issue, #18684:
URL: https://github.com/apache/datafusion/issues/18684

   If a limit is set for FileScanConfig, I wonder if it's reasonable to also set the batch_size to min(batch_size, limit), which might reduces the number of rows being scanned and decoded.
   
   Originally posted https://github.com/apache/arrow-rs/issues/8836


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Question: should set FileScanConfig's batch_size to min(batch_size, limit) [datafusion]

Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on issue #18684:
URL: https://github.com/apache/datafusion/issues/18684#issuecomment-3530722901

   This is a good point, we may can add some unit test for confirm it will scan more data if we have limit.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Discussion: should set FileScanConfig's batch_size to min(batch_size, limit) [datafusion]

Posted by "niebayes (via GitHub)" <gi...@apache.org>.
niebayes commented on issue #18684:
URL: https://github.com/apache/datafusion/issues/18684#issuecomment-3531262565

   Generally, the default batch size is relatively large (8192) compared to the limit (typically, 1 ~ 10). I think this optimization would introduce considerable performance.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Discussion: should set FileScanConfig's batch_size to min(batch_size, limit) [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #18684:
URL: https://github.com/apache/datafusion/issues/18684#issuecomment-3533180706

   I suspect for sources that already handle the limit specially (like Parquet reader) it won't change behavior very much
   
   However, for sources that don't have special limit handling, this change would improve things


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org