You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "TheR1sing3un (via GitHub)" <gi...@apache.org> on 2026/07/15 03:39:21 UTC
[PR] [python] Support read.parallelism=auto for single-process reads [paimon]
TheR1sing3un opened a new pull request, #8637:
URL: https://github.com/apache/paimon/pull/8637
### Purpose
`read.parallelism` defaulted to 1 (serial), the bottleneck for multi-split `to_arrow`/`to_pandas`/`to_duckdb` reads done directly through pypaimon.
This adds an opt-in `"auto"` value that resolves to `min(number of splits, CPU count)`, accepted both as the `read.parallelism` option and as the runtime `parallelism` argument. The global default stays 1 so distributed engines (Ray/Daft), which already parallelize across splits at the outer layer, do not double-parallelize.
`READ_PARALLELISM` becomes a string option to carry `"auto"`; on-disk table options are already stored as strings, so existing int values keep working and cross-engine schemas are unaffected.
### Tests
New unit tests for parallelism parsing (`auto`, integer strings, invalid values, case/whitespace) plus end-to-end `auto`-vs-serial parity tests via both the runtime argument and the table option in `reader_parallel_test.py`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Support read.parallelism=auto for single-process reads [paimon]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on PR #8637:
URL: https://github.com/apache/paimon/pull/8637#issuecomment-4988800300
> -1 This looks not good. If we want auto. None is auto.
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Support read.parallelism=auto for single-process reads [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #8637:
URL: https://github.com/apache/paimon/pull/8637
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org