You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JingsongLi (via GitHub)" <gi...@apache.org> on 2026/09/23 13:54:05 UTC

[PR] [rust] Account writer memory with ResourceContext [paimon-rust]

JingsongLi opened a new pull request, #928:
URL: https://github.com/apache/paimon-rust/pull/928

   ### Purpose
   
   Table writers currently bypass `ResourceContext`, so buffered write data is invisible to a caller's memory budget. This change connects the write path to the same resource context used by readers, including the DataFusion task memory pool.
   
   ### Brief change log
   
   - Add optional `with_resources` configuration to table write builders and propagate it through append, key-value, dedicated-format, and postpone writers.
   - Reserve estimated Arrow batch memory while format writers retain data, and reserve key-value batches until they are flushed.
   - Make write and commit-preparation failures terminal and clean up uncommitted files, including outputs from other partitions that prepared successfully.
   
   ### Tests
   
   - `cargo test -p paimon --test writer_resources_test` (10 passed)
   - `cargo test -p paimon --lib inference_buffer_is_charged_without_triggering_row_group_flush`
   - `cargo test -p paimon --lib test_postpone_fixed_bucket_builder_modes`
   - Existing table-write tests for append, partitioned, blob, and reusable prepare-commit paths passed.
   - `cargo check -p paimon-datafusion`, `cargo fmt --all -- --check`, and `git diff --check` passed.
   
   Release-mode write microbenchmark on the in-memory backend: each run wrote 128 prebuilt batches of 8,192 rows (1,048,576 rows) and called `prepare_commit`. Nine measured runs per variant followed two warmups; variants alternated. The comparison isolates the optional resource accounting on the same code revision.
   
   | Write mode | Without resources | With resources | Median delta |
   | --- | ---: | ---: | ---: |
   | Append | 28.360 ms | 28.242 ms | -0.42% |
   | Primary key | 70.430 ms | 70.975 ms | +0.77% |
   
   The measured differences are within run-to-run variation for this workload.
   
   ### API and Format
   
   Adds an optional writer builder API; no storage-format change.
   
   ### Documentation
   
   Updates `ResourceContext` documentation to describe write-memory accounting.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [rust] Account writer memory with ResourceContext [paimon-rust]

Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #928:
URL: https://github.com/apache/paimon-rust/pull/928#discussion_r4083920366


##########
crates/paimon/src/arrow/format/mod.rs:
##########
@@ -137,6 +143,68 @@ pub(crate) trait FormatFileWriter: Send {
     async fn close(self: Box<Self>) -> crate::Result<FormatWriteResult>;
 }
 
+/// Account for batches retained by a format writer until its next flush.
+/// The charge is an estimate of input Arrow buffers, not encoded allocations.
+pub(crate) fn with_write_resources(
+    writer: Box<dyn FormatFileWriter>,
+    resources: Option<&crate::resource::ResourceContext>,
+) -> Box<dyn FormatFileWriter> {
+    match resources {
+        Some(resources) => Box::new(ResourceFormatWriter {
+            inner: writer,
+            reservation: resources.reservation(),
+        }),
+        None => writer,
+    }
+}
+
+struct ResourceFormatWriter {
+    inner: Box<dyn FormatFileWriter>,
+    reservation: crate::resource::MemoryReservation,
+}
+
+#[async_trait]
+impl FormatFileWriter for ResourceFormatWriter {
+    async fn write(&mut self, batch: &RecordBatch) -> crate::Result<()> {
+        self.reservation.try_grow(batch.get_array_memory_size())?;
+        self.inner.write(batch).await?;
+        if !self.inner.retains_batch_data() {
+            self.reservation.try_resize(0)?;
+        }

Review Comment:
   [P2] Release the charge for row groups that Parquet flushes internally
   
   Parquet can flush a full row group inside write() and leave a small tail in the next group. In that case retains_batch_data() stays true, so this code retains the charge for all previously flushed input batches as well. The reservation keeps growing across internal flushes until an entirely empty buffer, an explicit flush, or file close, and valid writes can fail with ResourceExhausted even though almost all of the charged data has already been flushed. This also affects the DataFusion sink, which now always enables this accounting.
   
   Reproduced on this head with an append table containing two Int32 columns, file.format=parquet, target-file-size=1gb, and write.parquet-buffer-size=64mb. Write the same 1,048,577-row batch three times under a 20,972,020-byte budget (2.5 times the batch's 8,388,808-byte Arrow estimate). Parquet's default 1,048,576-row boundary flushes a full group on each call, but reservations after the first two writes are 8,388,808 and 16,777,616 bytes; the third write fails while requesting another 8,388,808 bytes. Without resources all three writes succeed, and the footer confirms row groups [1048576, 1048576, 1048576, 3]. With resources and exactly 1,048,576 rows per batch, all three writes also succeed and the reservation returns to zero after each write.
   
   Please reconcile the charge with the data still retained after internal format flushes, releasing the emitted groups' reservation while preserving the unfinished group's charge, and add coverage for non-row-group-aligned batches.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [rust] Account writer memory with ResourceContext [paimon-rust]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #928:
URL: https://github.com/apache/paimon-rust/pull/928


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org