Re: [PR] [datafusion] Honor Paimon read.batch-size [paimon-rust]

["leaves12138 (via GitHub)" <gi...@apache.org>]

leaves12138 commented on PR #547:
URL: https://github.com/apache/paimon-rust/pull/547#issuecomment-5016078482

   There is still one semantics gap in the current implementation: a non-default `read.batch-size` is not propagated into the composite primary-key and data-evolution readers.
   
   - `KeyValueFileReader` constructs `DataFileReader` in `crates/paimon/src/table/kv_file_reader.rs` without calling `with_batch_size`.
   - `DataEvolutionReader` does the same for its direct and column-source reads in `crates/paimon/src/table/data_evolution_reader.rs`.
   
   Those paths therefore pass `None` to the format reader, so Parquet continues to decode with its default batch size of 1024. The PR note says that PK merge and data-evolution *output* batching keeps its existing boundaries, but currently the input file-decoder batch size also ignores the table option. Java Paimon injects `READ_BATCH_SIZE` when constructing the `FileFormat`, so the underlying format readers receive it regardless of the higher-level merge reader.
   
   Please thread the resolved batch size through `KeyValueReadConfig` / `KeyValueFileReader` and the `DataEvolutionReader` file sources. The final merge output can still keep `SortMergeReaderBuilder`'s 1024 boundary and `MERGE_BATCH_SIZE = 1024` if that is intentional. A regression test using a non-default value on a merge-required PK split and a data-evolution read would cover both paths.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org