You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "kevinjqliu (via GitHub)" <gi...@apache.org> on 2026/09/15 15:47:04 UTC
[PR] Increase the default PyArrow IO buffer to 8 MiB [iceberg-python]
kevinjqliu opened a new pull request, #3971:
URL: https://github.com/apache/iceberg-python/pull/3971
# Rationale for this change
Raise the PyArrow FileIO buffer default from 1 MiB to 8 MiB. Use the same default for input/output streams and the Parquet scan and positional-delete readers. Regular Parquet scans already specified 8 MiB.
8 MiB is a reasonable starting point for sequential object-store reads. [Hadoop’s S3 prefetcher](https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/prefetching.html) and [Trino’s Parquet reader](https://github.com/trinodb/trino/blob/ea6ca09ae29eea442e01a79bbc650987e8897ec1/lib/trino-parquet/src/main/java/io/trino/parquet/ParquetReaderOptions.java#L25-L29) use that size for read blocks. These are precedents, not evidence of a measured speedup here.
## Are these changes tested?
Added assertions for the constructor default and both FileIO factories, with and without a buffer-size override. Focused PyArrow and Avro tests pass.
## Are there any user-facing changes?
The default stream buffer increases to 8 MiB for reads and writes, including output files converted to input files. Existing `buffer-size` overrides still work. Seekable reads and Arrow’s S3 multipart upload size are unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Increase the default PyArrow IO buffer to 8 MiB [iceberg-python]
Posted by "rambleraptor (via GitHub)" <gi...@apache.org>.
rambleraptor commented on code in PR #3971:
URL: https://github.com/apache/iceberg-python/pull/3971#discussion_r4017860901
##########
pyiceberg/io/pyarrow.py:
##########
@@ -195,8 +195,8 @@
logger = logging.getLogger(__name__)
-ONE_MEGABYTE = 1024 * 1024
BUFFER_SIZE = "buffer-size"
+_DEFAULT_BUFFER_SIZE = 8 * 1024 * 1024 # 8 MiB
Review Comment:
This doesn't really need to be a private variable. None of the others are.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Increase the default PyArrow IO buffer to 8 MiB [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3971:
URL: https://github.com/apache/iceberg-python/pull/3971#issuecomment-6355387967
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Increase the default PyArrow IO buffer to 8 MiB [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3971:
URL: https://github.com/apache/iceberg-python/pull/3971#issuecomment-6205191163
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Increase the default PyArrow IO buffer to 8 MiB [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #3971: Increase the default PyArrow IO buffer to 8 MiB
URL: https://github.com/apache/iceberg-python/pull/3971
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Increase the default PyArrow IO buffer to 8 MiB [iceberg-python]
Posted by "rambleraptor (via GitHub)" <gi...@apache.org>.
rambleraptor commented on PR #3971:
URL: https://github.com/apache/iceberg-python/pull/3971#issuecomment-5684133047
The local benchmarks you did look great. It looks like a huge speed up on larger cases with basically no impact on the smaller cases. (0.2ms is just small enough where I think there's the possibility of some noise there)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org