Re: [PR] [python] Pipeline split reads in Arrow batch reader [paimon]

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi commented on PR #8807:
URL: https://github.com/apache/paimon/pull/8807#issuecomment-5167040499

   Thanks for working on this. I suggest making the default parallelism adaptive instead of enabling `min(number of splits, CPU count)` for every multi-split read.
   
   The pipeline should help when splits are sufficiently large or backed by high-latency remote storage such as OSS/S3/HDFS. However, thread startup, per-split queues, and ordered delivery can increase time-to-first-batch and memory usage for small splits or local/cached files.
   
   Could we choose the default path based on estimated split bytes and storage characteristics? For example:
   
   - stay serial for small total input or local/low-latency storage;
   - increase or cap parallelism for larger inputs or high-latency object stores;
   - keep the explicit `parallelism` argument and `read.parallelism` option as authoritative overrides.
   
   A byte-based readahead budget, together with a benchmark matrix covering split size, storage backend, parallelism, time-to-first-batch, throughput, and peak RSS, would also make the heuristic safer.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org