You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "gabrnavarro (via GitHub)" <gi...@apache.org> on 2026/04/03 20:35:32 UTC

[PR] Fix: fix: read Parquet batch size from table properties instead of hardcoding [iceberg-go]

gabrnavarro opened a new pull request, #849:
URL: https://github.com/apache/iceberg-go/pull/849

   ## What changed
   
   - Added `ParquetBatchSizeKey` (`read.parquet.batch-size`) and `ParquetBatchSizeDefault` (131072) constants to the internal and public properties packages.
   - Added `WithTableProperties` / `TablePropertiesFromContext` context helpers in `table/internal` to carry table properties through the read path.
   - `arrowScan.GetRecords` injects the table's `Metadata.Properties()` into the context before any file reads (data files and delete files).
   - `ParquetFileSource.GetReader` reads `read.parquet.batch-size` from context properties instead of hardcoding `1 << 17`.
   
   ## Why
   
   `parquet_files.go` hardcoded `BatchSize: 1 << 17` with a TODO comment to grab it from context. The write path already reads page row limit from table properties via `ParquetPageRowLimitKey`; this PR brings the read path into alignment.
   
   Fixes #846
   
   ## Testing
   
   - Added `TestParquetBatchSizeFromTableProperties` in `table/internal/parquet_files_test.go` covering default, custom, and invalid values.
   - All existing unit tests pass (`go test ./...`).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix(table): read Parquet batch size from table properties [iceberg-go]

Posted by "gabrnavarro (via GitHub)" <gi...@apache.org>.
gabrnavarro commented on PR #849:
URL: https://github.com/apache/iceberg-go/pull/849#issuecomment-4187318221

   All requested changes have been addressed:
   - Title updated to `fix(table): read Parquet batch size from table properties`
   - Linter passes (`golangci-lint run ./...` with v2.8.0 — 0 issues)
   - All CI checks green
   
   Ready for re-review when you get a chance!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix(table): read Parquet batch size from table properties [iceberg-go]

Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on PR #849:
URL: https://github.com/apache/iceberg-go/pull/849#issuecomment-4187438023

   But what close? :D


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix(table): read Parquet batch size from table properties [iceberg-go]

Posted by "gabrnavarro (via GitHub)" <gi...@apache.org>.
gabrnavarro commented on PR #849:
URL: https://github.com/apache/iceberg-go/pull/849#issuecomment-4187435259

   Closing this PR due to inactivity. Thank you for the opportunity to contribute!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Fix: fix: read Parquet batch size from table properties instead of hardcoding [iceberg-go]

Posted by "gabrnavarro (via GitHub)" <gi...@apache.org>.
gabrnavarro commented on PR #849:
URL: https://github.com/apache/iceberg-go/pull/849#issuecomment-4187279259

   Good catch, updated the title to just 'Read Parquet batch size from table properties instead of hardcoding'


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix(table): read Parquet batch size from table properties [iceberg-go]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #849:
URL: https://github.com/apache/iceberg-go/pull/849#issuecomment-4189234233

   Why close this? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Fix: fix: read Parquet batch size from table properties instead of hardcoding [iceberg-go]

Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on PR #849:
URL: https://github.com/apache/iceberg-go/pull/849#issuecomment-4187295048

   PR-title is still `Fix: fix: read Parquet batch size from table properties instead of hardcoding`
   
   Generally speaking we lean towards prefix-based approach
   
   type(area): description
   
   In your case it would be:
   
   ```
   fix(table): read Parquet batch size from table properties
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix(table): read Parquet batch size from table properties [iceberg-go]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade merged PR #849:
URL: https://github.com/apache/iceberg-go/pull/849


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix(table): read Parquet batch size from table properties [iceberg-go]

Posted by "gabrnavarro (via GitHub)" <gi...@apache.org>.
gabrnavarro closed pull request #849: fix(table): read Parquet batch size from table properties
URL: https://github.com/apache/iceberg-go/pull/849


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix(table): read Parquet batch size from table properties [iceberg-go]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #849:
URL: https://github.com/apache/iceberg-go/pull/849#issuecomment-4189235999

   I'm gonna reopen this because this is a good contribution!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Fix: fix: read Parquet batch size from table properties instead of hardcoding [iceberg-go]

Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on PR #849:
URL: https://github.com/apache/iceberg-go/pull/849#issuecomment-4187270781

   also - plz change title, fix state there twice


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org