You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "jianguotian (via GitHub)" <gi...@apache.org> on 2026/08/07 12:17:40 UTC

[PR] [core] Add direct write path for BundleRecords [paimon]

jianguotian opened a new pull request, #9100:
URL: https://github.com/apache/paimon/pull/9100

   ### Purpose
   
   Writing `BundleRecords` currently requires converting a vector batch into `InternalRow` objects and processing the rows one by one, even when the reader and format writer support the same vector representation.
   
   This PR introduces a format-neutral, opt-in contract for synchronously passing borrowed vector batches to compatible format writers. Existing bundle implementations continue to use the row-by-row path by default, so current Paimon behavior is unchanged.
   
   ### Changes
   
   - Add `BundleRecords.isDirectWriteBundle()` as an opt-in capability; its default value is `false`.
   - Expose borrowed Arrow batches through `ArrowVectorizedRecordIterator` and carry the final output `RowType` in `ArrowBundleRecords`.
   - Add `ArrowBatchReader.readVectorizedBatch()` for independent batch wrappers while preserving wrapper reuse in the existing `readBatch()` path.
   - Add a guarded direct-bundle path in `RowDataFileWriter` only when there are no auxiliary writers, row tracking, or statistics that require per-record collection.
   - Update generated sequence numbers using the exact bundle row count after a direct write.
   - Preserve row-by-row fallback for unmarked bundles, per-record statistics, row tracking, and file indexes.
   - Document the borrowed Arrow-buffer lifetime: the writer must consume the bundle synchronously before `writeBundle()` returns.
   
   ### Tests
   
   - `ArrowBatchReaderTest`: 2 tests passed, covering reusable row-batch wrappers and independent vectorized-batch wrappers.
   - `RowDataFileWriterTest`: 6 tests passed, covering the direct path, extractor-based statistics, and all guarded fallback paths.
   - Maven Checkstyle, Spotless, and Enforcer checks passed in the focused reactor build.
   - `git diff --check` passed.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core][format] Support Arrow bundle writes [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #9100:
URL: https://github.com/apache/paimon/pull/9100


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core][format] Support Arrow bundle writes [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #9100:
URL: https://github.com/apache/paimon/pull/9100#issuecomment-5276641047

   +1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Add direct write path for BundleRecords [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #9100:
URL: https://github.com/apache/paimon/pull/9100#issuecomment-5218243238

   Thanks for working on this. I wonder whether `BundleRecords.isDirectWriteBundle()` is needed at all.
   
   As I understand it, this flag does not actually guarantee a zero-copy or native write. It only controls whether `RowDataFileWriter` may bypass its per-record path and forward the bundle to `SingleFileWriter`. Once the per-record consumers have been ruled out (auxiliary writers, row tracking, and per-record statistics), it should be safe to forward any `BundleRecords`:
   
   ```java
   long rowCount = bundle.rowCount();
   if (auxiliaryFileWriters.isEmpty()
           && sequenceNumberTracker.supportsRowCountUpdate()
           && !requiresPerRecordStats()) {
       super.writeBundle(bundle);
       sequenceNumberTracker.updateByRowCount(rowCount);
       return;
   }
   
   for (InternalRow row : bundle) {
       write(row);
   }
   ```
   
   The existing abstractions already provide the remaining dispatch:
   
   - `SingleFileWriter.writeBundle` falls back to row iteration when the format writer is not a `BundleFormatWriter`.
   - A `BundleFormatWriter` can inspect the bundle type and decide whether to consume it natively, convert or copy it, or fall back to `addElement` row by row.
   - If the underlying native write is asynchronous, the format writer can copy the borrowed data or acquire independent ownership before returning.
   
   In other words, whether to copy or use a native batch path looks like a format-writer implementation detail. The extra boolean duplicates that dispatch and is slightly misleading: `true` still does not mean that the write is direct or zero-copy.
   
   Could we remove `isDirectWriteBundle()` and instead strengthen the contract of `BundleFormatWriter.writeBundle()` to require semantic equivalence with row-by-row writes and prohibit retaining borrowed buffers after the method returns unless they have been copied or independently retained? This would keep `BundleRecords` as a data abstraction and leave the write strategy entirely to the format writer.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [mosaic] Support Arrow bundle writes [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #9100:
URL: https://github.com/apache/paimon/pull/9100#issuecomment-5236053377

   We should remove `supportsRowEquivalentBundleWrite` too.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core][format] Support Arrow bundle writes [paimon]

Posted by "jianguotian (via GitHub)" <gi...@apache.org>.
jianguotian commented on PR #9100:
URL: https://github.com/apache/paimon/pull/9100#issuecomment-5240905161

   > undleRecords.isDirectWriteBundle()
   
   Both removed.
   Since these writeBundle methods can now be reached through the general RowDataFileWriter path, I also hardened the Arrow, Lance, and Vortex implementations for schema/layout compatibility, pending-row ordering, allocator and buffer lifetime, and failure cleanup. Incompatible bundles fall back to row writes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org