You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "qzyu999 (via GitHub)" <gi...@apache.org> on 2026/08/20 01:51:44 UTC

[PR] [lake/iceberg] Implement SupportsRecordBatchWrite for IcebergLakeWriter [fluss]

qzyu999 opened a new pull request, #4048:
URL: https://github.com/apache/fluss/pull/4048

   ## Summary
   
   Implements SupportsRecordBatchWrite for the Iceberg lake writer, enabling direct Arrow batch writing for append-only log tables during tiering. This closes the performance gap with Paimon's batch write path (landed in #3417/#3418).
   
   Closes #4047
   
   ## Changes
   
   - **IcebergLakeWriter** now implements SupportsRecordBatchWrite
   - New **IcebergArrowBatchHelper** reads Arrow FieldVector columns directly and produces Iceberg GenericRecord objects enriched with system columns (__bucket, __offset, __timestamp), then writes through the existing TaskWriter<Record> pipeline
   - Added rrow-vector and rrow-memory-netty as provided dependencies (same pattern as luss-lake-paimon)
   - Only enabled for append-only tables (same restriction as Paimon)
   
   ## Motivation
   
   The Paimon lake writer implements SupportsRecordBatchWrite which takes raw Arrow batches from Fluss's internal format and writes them to Parquet via paimon-arrow. The Iceberg path previously wrote row-by-row through GenericRecord objects wrapping LogRecord, requiring per-record binary deserialization.
   
   With this change, the Iceberg tiering path can read directly from Arrow columnar memory  avoiding the LogRecord  InternalRow  FlussRecordAsIcebergRecord conversion chain for each record.
   
   ## Design
   
   The helper creates typed ArrowFieldExtractor lambdas per column based on the Fluss DataType, then iterates rows reading from the pre-fetched FieldVector array. This preserves the existing TaskWriter pipeline (file rolling, partition key generation, Iceberg metrics tracking) while eliminating the input-side deserialization cost.
   
   Supported types: boolean, tinyint, smallint, int, bigint, float, double, string, char, binary, bytes, decimal, date, time, timestamp, timestamp_ltz. Complex types (array, map, row) have placeholder extractors and are left as follow-up work.
   
   ## Test Plan
   
   - Compilation verified: mvn compile -pl fluss-lake/fluss-lake-iceberg -am -DskipTests
   - Formatting: mvn spotless:check passes
   - Checkstyle: mvn checkstyle:check passes
   - Existing tests: IcebergLakeCatalogTest, IcebergWriteResultSerializerTest, FlussToIcebergPredicateConverterTest, IcebergConfigurationTest all pass
   - Integration test coverage needed: a dedicated test writing ArrowRecordBatch through the new path (follow-up)
   
    AI-assisted changes - reviewed by human developer


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [lake/iceberg] Implement SupportsRecordBatchWrite for IcebergLakeWriter [fluss]

Posted by "mikepapadim (via GitHub) via issues" <is...@fluss.apache.org>.
mikepapadim commented on PR #4048:
URL: https://github.com/apache/fluss/pull/4048#issuecomment-6880604819

   Thanks for the pointer. I benchmarked apache/parquet-java#3734 on the data from the tiering profile: TPC-H lineitem-like, 19 columns with Fluss's system columns, 16384-row Arrow batches, ZSTD. I built it against parquet 1.18.2. As is, it writes 512 MB in 2.57 s, versus 7.3 s for Fluss's Iceberg row path and 3.2 s for the Paimon writer. I also found a page-sizing issue in it. With that fixed, it takes 1.73 s and the files are 19% smaller. I've left the details on apache/parquet-java#3734 and opened the fix against your branch (https://github.com/qzyu999/parquet-java/pull/1). The Fluss-side numbers are in https://github.com/apache/fluss/discussions/5227.
   
   On the Fluss side, #5223 makes the batch path reachable for Iceberg, and #5226 proposes a way to plug in an Arrow file writer. apache/parquet-java#3734 would be the obvious CPU implementation for it, so #4048 wouldn't have to wait on a parquet-java release to get the columnar path wired up.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [lake/iceberg] Implement SupportsRecordBatchWrite for IcebergLakeWriter [fluss]

Posted by "github-actions[bot] (via GitHub) via issues" <is...@fluss.apache.org>.
github-actions[bot] commented on PR #4048:
URL: https://github.com/apache/fluss/pull/4048#issuecomment-6599090972

   This PR has been marked as stale because it has had no activity for 60 days. It will be closed in 10 days unless it receives a new comment, commit or other update.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [lake/iceberg] Implement SupportsRecordBatchWrite for IcebergLakeWriter [fluss]

Posted by "mikepapadim (via GitHub) via issues" <is...@fluss.apache.org>.
mikepapadim commented on PR #4048:
URL: https://github.com/apache/fluss/pull/4048#issuecomment-6859447181

   @qzyu999 Heads-up: with this PR alone, Iceberg tables would still be tiered row by row. `TieringSplitReader#useRecordBatchPath` only takes the `SupportsRecordBatchWrite` route when the lake format is Paimon.
   
   I've opened #5223 (issue #5222) to let the tiering factory declare it instead, through `LakeTieringFactory#supportsRecordBatchWrite(TableInfo)`. It's about 20 lines and leaves Paimon unchanged. Once that's in, `IcebergLakeTieringFactory` can return true for the tables this PR supports. Separately, this PR targets `fluss-lake-iceberg`, which has since been split into `fluss-lake-iceberg-common` and `fluss-lake-iceberg-1.12`, so it will need a rebase.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [lake/iceberg] Implement SupportsRecordBatchWrite for IcebergLakeWriter [fluss]

Posted by "qzyu999 (via GitHub) via issues" <is...@fluss.apache.org>.
qzyu999 commented on PR #4048:
URL: https://github.com/apache/fluss/pull/4048#issuecomment-6611110075

   bump


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [lake/iceberg] Implement SupportsRecordBatchWrite for IcebergLakeWriter [fluss]

Posted by "qzyu999 (via GitHub) via issues" <is...@fluss.apache.org>.
qzyu999 commented on PR #4048:
URL: https://github.com/apache/fluss/pull/4048#issuecomment-6881749932

   Thanks for benchmarking this, @mikepapadim! I merged your page-sizing fix into the branch for apache/parquet-java#3734. For this PR, I’ll wait for #5223 to land, then rebase and wire up the Iceberg factory so append-only Arrow tables can use the batch path.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [lake/iceberg] Implement SupportsRecordBatchWrite for IcebergLakeWriter [fluss]

Posted by "qzyu999 (via GitHub) via issues" <is...@fluss.apache.org>.
qzyu999 commented on PR #4048:
URL: https://github.com/apache/fluss/pull/4048#issuecomment-6859528152

   Hi @mikepapadim, thanks for taking a look, unfortunately this looks heavily bottlenecked by https://github.com/apache/parquet-java/pull/3734, not sure if you're willing to take a look to help push it forward.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org