[I] write.parquet.page-row-limit has no effect: mapped to pyarrow's write_batch_size [iceberg-python]

["YuangGao (via GitHub)" <gi...@apache.org>]

YuangGao opened a new issue, #4126:
URL: https://github.com/apache/iceberg-python/issues/4126

   ### Apache Iceberg version
   
   main (development)
   
   ### Please describe the bug 🐞
   
   `write.parquet.page-row-limit` is mapped to pyarrow's `write_batch_size` ([pyarrow.py#L3002-L3006](https://github.com/apache/iceberg-python/blob/7d51e52bfd42dba2806979ee2554507e95fcfa0a/pyiceberg/io/pyarrow.py#L3002-L3006)), but `write_batch_size` is how many values pyarrow writes before checking the page size, not a per-page row limit. So the property has no effect.
   
   Two side effects:
   
   - The default of 20000 is always passed as `write_batch_size` (pyarrow's own default is 1024), so pages are size-checked far less often and can overshoot `write.parquet.page-size-bytes`.
   - Java maps this property to parquet-java's `withPageRowCountLimit`, so the same table writes differently from Java and PyIceberg.
   
   pyarrow 23.0.0 added `max_rows_per_page` (apache/arrow#48101), which is the actual per-page row limit.
   
   To reproduce, 50,000 int64 rows (column chunk size in bytes):
   
   ```python
   import pyarrow as pa
   import pyarrow.parquet as pq
   from pyiceberg.catalog.sql import SqlCatalog
   
   data = pa.table({"id": pa.array(range(50_000), pa.int64())})
   catalog = SqlCatalog("t", uri="sqlite:////tmp/cat.db", warehouse="file:///tmp/wh")
   catalog.create_namespace("ns")
   
   for name, props in [("a", {}), ("b", {"write.parquet.page-row-limit": "100"})]:
       table = catalog.create_table(f"ns.{name}", schema=data.schema, properties=props)
       table.append(data)
       path = next(iter(table.scan().plan_files())).file.file_path.removeprefix("file://")
       print(name, pq.ParquetFile(path).metadata.row_group(0).column(0).total_compressed_size)
   ```
   
   | | size |
   |---|---|
   | no properties | 145139 |
   | `write.parquet.page-row-limit=100` | 145139 (unchanged) |
   | pyarrow `max_rows_per_page=100`, for comparison | 181473 |
   
   Proposed fix: stop passing `write_batch_size`, and pass `max_rows_per_page` when pyarrow >= 23.0.0.
   
   ### Willingness to contribute
   
   - [x] I can contribute a fix for this bug independently
   - [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
   - [ ] I cannot contribute a fix for this bug at this time
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] write.parquet.page-row-limit has no effect: mapped to pyarrow's write_batch_size [iceberg-python]

["Fokko (via GitHub)" <gi...@apache.org>]

Fokko closed issue #4126: write.parquet.page-row-limit has no effect: mapped to pyarrow's write_batch_size
URL: https://github.com/apache/iceberg-python/issues/4126


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org