You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "gabrnavarro (via GitHub)" <gi...@apache.org> on 2026/04/03 20:35:32 UTC
[PR] Fix: fix: read Parquet batch size from table properties instead of hardcoding [iceberg-go]
gabrnavarro opened a new pull request, #849:
URL: https://github.com/apache/iceberg-go/pull/849
## What changed
- Added `ParquetBatchSizeKey` (`read.parquet.batch-size`) and `ParquetBatchSizeDefault` (131072) constants to the internal and public properties packages.
- Added `WithTableProperties` / `TablePropertiesFromContext` context helpers in `table/internal` to carry table properties through the read path.
- `arrowScan.GetRecords` injects the table's `Metadata.Properties()` into the context before any file reads (data files and delete files).
- `ParquetFileSource.GetReader` reads `read.parquet.batch-size` from context properties instead of hardcoding `1 << 17`.
## Why
`parquet_files.go` hardcoded `BatchSize: 1 << 17` with a TODO comment to grab it from context. The write path already reads page row limit from table properties via `ParquetPageRowLimitKey`; this PR brings the read path into alignment.
Fixes #846
## Testing
- Added `TestParquetBatchSizeFromTableProperties` in `table/internal/parquet_files_test.go` covering default, custom, and invalid values.
- All existing unit tests pass (`go test ./...`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): read Parquet batch size from table properties [iceberg-go]
Posted by "gabrnavarro (via GitHub)" <gi...@apache.org>.
gabrnavarro commented on PR #849:
URL: https://github.com/apache/iceberg-go/pull/849#issuecomment-4187318221
All requested changes have been addressed:
- Title updated to `fix(table): read Parquet batch size from table properties`
- Linter passes (`golangci-lint run ./...` with v2.8.0 — 0 issues)
- All CI checks green
Ready for re-review when you get a chance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): read Parquet batch size from table properties [iceberg-go]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on PR #849:
URL: https://github.com/apache/iceberg-go/pull/849#issuecomment-4187438023
But what close? :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): read Parquet batch size from table properties [iceberg-go]
Posted by "gabrnavarro (via GitHub)" <gi...@apache.org>.
gabrnavarro commented on PR #849:
URL: https://github.com/apache/iceberg-go/pull/849#issuecomment-4187435259
Closing this PR due to inactivity. Thank you for the opportunity to contribute!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix: fix: read Parquet batch size from table properties instead of hardcoding [iceberg-go]
Posted by "gabrnavarro (via GitHub)" <gi...@apache.org>.
gabrnavarro commented on PR #849:
URL: https://github.com/apache/iceberg-go/pull/849#issuecomment-4187279259
Good catch, updated the title to just 'Read Parquet batch size from table properties instead of hardcoding'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): read Parquet batch size from table properties [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #849:
URL: https://github.com/apache/iceberg-go/pull/849#issuecomment-4189234233
Why close this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix: fix: read Parquet batch size from table properties instead of hardcoding [iceberg-go]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on PR #849:
URL: https://github.com/apache/iceberg-go/pull/849#issuecomment-4187295048
PR-title is still `Fix: fix: read Parquet batch size from table properties instead of hardcoding`
Generally speaking we lean towards prefix-based approach
type(area): description
In your case it would be:
```
fix(table): read Parquet batch size from table properties
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): read Parquet batch size from table properties [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade merged PR #849:
URL: https://github.com/apache/iceberg-go/pull/849
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): read Parquet batch size from table properties [iceberg-go]
Posted by "gabrnavarro (via GitHub)" <gi...@apache.org>.
gabrnavarro closed pull request #849: fix(table): read Parquet batch size from table properties
URL: https://github.com/apache/iceberg-go/pull/849
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): read Parquet batch size from table properties [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #849:
URL: https://github.com/apache/iceberg-go/pull/849#issuecomment-4189235999
I'm gonna reopen this because this is a good contribution!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix: fix: read Parquet batch size from table properties instead of hardcoding [iceberg-go]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on PR #849:
URL: https://github.com/apache/iceberg-go/pull/849#issuecomment-4187270781
also - plz change title, fix state there twice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org