[I] [lake/iceberg] Implement SupportsRecordBatchWrite for IcebergLakeWriter [fluss]

["qzyu999 (via GitHub)" <gi...@apache.org>]

qzyu999 opened a new issue, #4047:
URL: https://github.com/apache/fluss/issues/4047

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
   
   ### Motivation
   
   Paimon's lake writer implements `SupportsRecordBatchWrite` (landed in #3417/#3418), enabling direct Arrow batch to Parquet writes for append-only tables. This avoids the row-by-row serialization overhead through `GenericRecord` objects.
   
   The Iceberg lake writer (`IcebergLakeWriter`) currently writes row-by-row via `GenericAppenderFactory`, resulting in significantly lower throughput for append-only log table tiering compared to the Paimon path.
   
   Since Fluss uses Arrow as its internal columnar format, and both Iceberg and Parquet have native Arrow support (`iceberg-arrow` module, `parquet-arrow` module), we can bridge the Arrow batch directly to Iceberg Parquet files.
   
   ### Solution
   
   1. Make `IcebergLakeWriter` implement `SupportsRecordBatchWrite`  
   2. Create an Arrow batch writer that takes `ArrowBatchData` and writes to Iceberg Parquet files using Parquet's Arrow write support
   3. Only enabled for append-only tables (same restriction as Paimon)  
   4. Preserve Iceberg file metrics tracking for proper table maintenance
   
   Related: #437 (generic arrow-to-parquet issue)
   
   ### Anything else?
   
   All necessary primitives exist in upstream Iceberg (`iceberg-arrow` 1.10.1) and Apache Parquet (`parquet-arrow`). No upstream changes needed.
   
   ### Willingness to contribute
   
   - [x] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [lake/iceberg] Implement SupportsRecordBatchWrite for IcebergLakeWriter [fluss]

["qzyu999 (via GitHub)" <gi...@apache.org>]

qzyu999 commented on issue #4047:
URL: https://github.com/apache/fluss/issues/4047#issuecomment-5362795970

   ## Upstream Context
   
   The current PR (#4048) implements SupportsRecordBatchWrite by reading Arrow vectors directly and producing Iceberg GenericRecord objects. This avoids binary log deserialization but still materializes row objects because **parquet-java lacks a batch/columnar write API**.
   
   For true zero-copy Arrow-to-Parquet writing (matching what Paimon achieves with paimon-arrow), the gap is in the underlying Parquet Java library:
   
   - **[apache/parquet-java#3733](https://github.com/apache/parquet-java/issues/3733)**  Filed to request batch write APIs + Arrow VectorSchemaRoot writer (Java parity with C++/Python write_table())
   - **[apache/parquet-java#2264](https://github.com/apache/parquet-java/issues/2264)**  Original request from 2019 (no activity)
   - **[apache/parquet-java#3530](https://github.com/apache/parquet-java/issues/3530)**  Active perf series that plans 'level write batching' as a future PR
   
   Once parquet-java exposes batch write methods on ValuesWriter, the Iceberg and Fluss layers can be updated to pass Arrow column buffers directly without row materialization.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org