You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "dwsmith1983 (via GitHub)" <gi...@apache.org> on 2026/09/28 17:08:03 UTC

[PR] perf: charge the local shuffle writer's buffers to the memory pool [datafusion-comet]

dwsmith1983 opened a new pull request, #6336:
URL: https://github.com/apache/datafusion-comet/pull/6336

   ## Which issue does this PR close?
   
   Closes #6196.
   
   ## Rationale for this change
   
   The local shuffle writer allocates buffers of `spark.comet.shuffle.native.writeBufferSize` bytes that no reservation covers: the data file `BufWriter`, the spill file `BufWriter`, the buffer `finish_partition` reads spilled ranges through, and the recycled scratch that blocks are encoded into, plus the zstd context when zstd is configured. With the 1 MiB default, a multi-partition task that has spilled holds about 4 MiB that neither Comet's pool nor Spark's `TaskMemoryManager` sees, so it has to fit in `spark.executor.memoryOverhead`.
   
   ## What changes are included in this PR?
   
   - `MultiPartitionShuffleRepartitioner::try_new` hands the writer a second reservation on the consumer it already registers, `reservation.new_empty()`, through a new `PartitionWriter::attach_buffer_reservation` with a no-op default. No new consumer, so `fair_unified`'s per-consumer share is unchanged, and the existing reservation, `maxBufferBytes` and `memory_spilled_bytes` are untouched: `spill()` still frees only the main reservation.
   - `LocalPartitionWriter` keeps that reservation equal to the bytes its buffers hold: the data file buffer capacity, the spill file buffer capacity, the copy buffer length, the scratch capacity and the zstd context size. It syncs at the end of `write`, `finish_partition`, `write_burst_complete` and `finish_all`, on every exit. `resize` makes no pool call when the size is unchanged, so steady state adds no pool traffic per batch; in practice it is one grow and one shrink per spill when zstd is on, plus the one-time buffer grows.
   - Buffers allocated up front ask for the full write buffer size with `try_grow`. When the pool refuses, the buffer is 8 KiB and that is what gets charged. The data file buffer is built before the consumer registers, so on a refusal it is rebuilt at the fallback size while still empty. The spill file and copy buffers are reserved after their file is open, so a failed open leaves no charge. Reads of a range larger than the copy buffer already go through `io::copy`.
   - Memory that already exists when it is charged (the scratch, the zstd context) uses `grow`, which Comet's pools record as overcommit. The scratch's transient peak between syncs, the write buffer size plus one block, stays uncharged.
   - The single-partition, empty-schema and remote shuffle writers are unchanged; they have no reservation to charge, and registering one would change `fair_unified`'s shares. The tuning guide and the native shuffle contributor doc list them as untracked.
   
   ## Benchmark
   
   A pressure-triggered spill in the shuffle crate: 200 hash partitions, a 16 MiB `GreedyMemoryPool`, 1 MiB write buffer, Lz4, 256 MiB of Int64 input in 8192-row batches, release build, medians of three runs.
   
   | | spill count | memory_spilled_bytes | spilled bytes | insert (spills) | final write | total |
   |---|---|---|---|---|---|---|
   | main | 36 | 607,136,768 | 153,994,145 | 692 ms | 25 ms | 717 ms |
   | this PR | 39 | 613,866,496 | 153,571,301 | 709 ms | 36 ms | 745 ms |
   
   The buffers take about 2 MiB of the 16 MiB pool, which is the three extra spills. The spill file buffer's grow runs during a pressure spill while the main reservation is still full, so it gets the 8 KiB fallback for the rest of the task, which is the slower final merge. Reserved bytes after the write: data buffer 1 MiB, spill buffer 8 KiB, copy buffer 1 MiB, scratch 38 KiB. Upgrading that buffer to full size right after a spill frees the main reservation, when the pool has room again, is a possible follow-up.
   
   ## How are these changes tested?
   
   Shuffle crate tests with a `GreedyMemoryPool`: the buffers are charged after attach, after a spill and after the write, the main reservation is empty after a spill, and the pool reads zero after drop; with pools of 1.5 MiB and 512 KiB the refused buffers fall back to 8 KiB, the reservation equals the bytes held after every step, and the output is byte-identical to a 1 GiB pool run; a refused copy buffer copies a range larger than it through `io::copy` and a smaller one through `pread`; with `maxBufferBytes` set, `spill_count` and `memory_spilled_bytes` match a run without buffer charging; with zstd, the reservation grows by the context size after a write and shrinks after `write_burst_complete` and `finish_all`; growing only the encode scratch moves the reservation by exactly the scratch's growth.
   
   The shuffle crate, the memory pool tests, clippy, fmt and `cargo bench --no-run` pass; `CometNativeShuffleSuite` and `CometShuffleSuite` pass on Spark 3.5 and `CometNativeShuffleSuite` on 4.1.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: charge the local shuffle writer's buffers to the memory pool [datafusion-comet]

Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #6336:
URL: https://github.com/apache/datafusion-comet/pull/6336#discussion_r4380775424


##########
native/shuffle/src/writers/local/mod.rs:
##########
@@ -17,3 +17,18 @@
 
 pub(crate) mod local_partition_writer;
 mod spill;
+
+use datafusion::execution::memory_pool::MemoryReservation;
+
+const FALLBACK_BUFFER_SIZE: usize = 8 * 1024;
+
+/// Returns the capacity to allocate. A refused size falls back to a small buffer charged with
+/// `grow`, so the task carries on and the reservation still matches what is held.
+fn reserve_buffer(reservation: &MemoryReservation, size: usize) -> usize {
+    if reservation.try_grow(size).is_ok() {

Review Comment:
   `reserve_buffer` and `restore_buffer` now fall back only when the root error is `ResourcesExhausted`. Any other error, such as the `External` a Spark-backed pool returns when a peer's spill throws, propagates from every buffer request and from the post-spill hook. `write_burst_complete` returns a `Result`, and `spill()` returns its error after freeing the reservation and recording the spill metrics, with a write error taking precedence. `buffer_pool_errors_other_than_refusal_fail_the_shuffle` runs the repartitioner on a pool that throws for the buffer requests and expects the shuffle to fail with nothing left reserved; `reserve_buffer_propagates_other_pool_errors`, `restore_buffer_returns_errors_other_than_refusal` and `spill_returns_burst_error_after_releasing_memory` cover the pieces.
   
   I left the `resize` in `sync_buffer_reservation` as it is. Its grow goes through `SparkMemory::acquire`, which on main turns a Java exception into a zero grant plus overcommit debt instead of an error. Every later `try_grow` asks Spark for that debt as well, so a repeated exception now surfaces from the next buffer request, and whatever was booked is released when the repartitioner drops. A charge booked from one swallowed exception can still sit there until then. The repartitioner's own batch reservation also treats any `try_grow` error as a refusal. Both paths are on main, so they are in #7244.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: charge the local shuffle writer's buffers to the memory pool [datafusion-comet]

Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #6336:
URL: https://github.com/apache/datafusion-comet/pull/6336#discussion_r4380872689


##########
native/shuffle/src/writers/local/spill.rs:
##########
@@ -214,17 +217,63 @@ impl PartitionedSpill {
     fn ensure_spill_file_created(
         &mut self,
         runtime: &RuntimeEnv,
+        buffers: Option<&MemoryReservation>,
     ) -> datafusion::common::Result<()> {
         if self.spill_file.is_none() {
             let temp_file = runtime
                 .disk_manager
                 .create_tmp_file("shuffle writer spill")?;
-            let writer = BufWriter::with_capacity(self.write_buffer_size, temp_file.open_writer()?);
+            // Open before charging, so a failed open leaves nothing on the reservation.
+            let file = temp_file.open_writer()?;
+            let capacity = buffers.map_or(self.write_buffer_size, |reservation| {
+                super::reserve_buffer(reservation, self.write_buffer_size)
+            });
+            let writer = BufWriter::with_capacity(capacity, file);
             self.spill_file = Some(ActiveSpillFile { temp_file, writer });
         }
         Ok(())
     }
 
+    /// Grows a fallback-sized buffer back to the configured size if `buffers` grants it.
+    /// Buffered bytes move to the new buffer; nothing is written to the file.
+    pub(crate) fn restore_buffer(&mut self, buffers: &MemoryReservation) {
+        if self.failed {
+            return;
+        }
+        let Some(spill_file) = self.spill_file.take() else {
+            return;
+        };
+        let capacity = spill_file.writer.capacity();
+        if capacity >= self.write_buffer_size
+            || buffers.try_grow(self.write_buffer_size - capacity).is_err()

Review Comment:
   After a spill the writer now grows the spill buffer to at most `min(write buffer size, freed / 2)`, where `freed` is what the spill released, and never shrinks it. That leaves at least half of the freed memory to the next batches. `spill_buffer_restore_leaves_room_for_the_input` runs your sweep (8 hash partitions, LZ4, no buffer limit, 1 MiB write buffer, 256 batches of 1,024 integers) at 2,112 KiB and 1,088 KiB. It checks that no restore grows the buffer past half of what that spill freed, that the writer spills at most about twice as often as a writer that holds no buffers, and that all rows decode.
   
   One edge remains: a spill buffer the pool grants in full when the first spill opens the file keeps that size, since the restore never shrinks it.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: charge the local shuffle writer's buffers to the memory pool [datafusion-comet]

Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #6336:
URL: https://github.com/apache/datafusion-comet/pull/6336#issuecomment-5881696182

   > Could those two passages call out the shuffle writer's zstd context as the exception, alongside the list this PR already updates higher up in the same tuning page?
   
   Done in 0d23cb4. The "Non-Rust allocations" bullet in `memory_management.md` now says the pool tracks such memory only where an operator charges it by hand, and names the multi-partition shuffle writer's zstd context as the one case today, in `reserved` and never in `allocated`. Step 3 of the sizing section in `tuning/memory.md` says the same, and that the `allocated - reserved` difference understates what has to fit in the overhead by one context per such task, with the two sizes you gave.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: charge the local shuffle writer's buffers to the memory pool [datafusion-comet]

Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #6336:
URL: https://github.com/apache/datafusion-comet/pull/6336#issuecomment-6044622342

   @andygrove when you get a chance, could you review this and approve the CI run? The branch is synced with `main`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: charge the local shuffle writer's buffers to the memory pool [datafusion-comet]

Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #6336:
URL: https://github.com/apache/datafusion-comet/pull/6336#discussion_r4380899805


##########
native/shuffle/src/writers/local/local_partition_writer.rs:
##########
@@ -245,88 +313,100 @@ impl PartitionWriter for LocalPartitionWriter {
         let write_buffer_size = self.write_buffer_size;
         let batch_size = self.batch_size;
 
-        match &mut self.data_output {
-            DataOutput::Single { writer, scratch } => {
-                // Single-partition data was already streamed via `write`, starting at
-                // offset 0 (already recorded in `self.offsets[0]`). Stream any trailing
-                // batches (normally none) without flushing; the long-lived writer is
-                // flushed once in `finish_all`.
-                for batch in iter.by_ref() {
-                    let batch = batch?;
-                    writer.write(
-                        &batch,
-                        scratch,
-                        &mut self.codec_context,
-                        &metrics.encode_time,
-                        &metrics.write_time,
-                    )?;
-                }
-            }
-            DataOutput::Multi {
-                output_writer,
-                shuffle_block_writer,
-                spill,
-                spill_reader,
-                recycled_buffer,
-                ..
-            } => {
-                self.offsets[pid] = output_writer.stream_position()?;
-
-                let mut flush_timer = metrics.write_time.timer();
-                spill.flush()?;
-                flush_timer.stop();
-                let ranges = spill.ranges(pid)?;
-                if !ranges.is_empty() {
-                    if spill_reader.is_none() {
-                        let path = spill.path()?.ok_or_else(|| {
-                            DataFusionError::Internal(
-                                "shuffle spill ranges recorded without a spill file".to_string(),
-                            )
-                        })?;
-                        *spill_reader = Some((File::open(path)?, vec![0; write_buffer_size]));
-                    }
-                    let (spill_file, buffer) = spill_reader.as_mut().unwrap();
-                    let mut write_timer = metrics.write_time.timer();
-                    for range in ranges {
-                        copy_spill_range(spill_file, buffer, range, output_writer)?;
-                    }
-                    write_timer.stop();
-                }
-
-                // Write in memory batches to output data file. Each partition uses its
-                // own writer so coalescing does not cross partition boundaries, but the
-                // scratch buffer is shared so its capacity carries over to the next one.
-                let mut buf_batch_writer = BufBatchWriter::new(
-                    shuffle_block_writer,
-                    output_writer,
-                    write_buffer_size,
-                    batch_size,
-                );
-                let codec_context = &mut self.codec_context;
-                let result: datafusion::common::Result<()> = (|| {
+        // Synced on every exit, errors included, so it matches what a failed partition left.
+        let result: datafusion::common::Result<()> = (|| {
+            match &mut self.data_output {
+                DataOutput::Single { writer, scratch } => {
+                    // Single-partition data was already streamed via `write`, starting at
+                    // offset 0 (already recorded in `self.offsets[0]`). Stream any trailing
+                    // batches (normally none) without flushing; the long-lived writer is
+                    // flushed once in `finish_all`.
                     for batch in iter.by_ref() {
                         let batch = batch?;
-                        buf_batch_writer.write(
+                        writer.write(
                             &batch,
-                            recycled_buffer,
-                            codec_context,
+                            scratch,
+                            &mut self.codec_context,
                             &metrics.encode_time,
                             &metrics.write_time,
                         )?;
                     }
-                    buf_batch_writer.flush(
-                        recycled_buffer,
-                        codec_context,
-                        &metrics.encode_time,
-                        &metrics.write_time,
-                    )
-                })();
-                // An errored partition must hand back a drained buffer, or its bytes
-                // leak into the next partition's block.
-                result.inspect_err(|_| recycled_buffer.clear())?;
+                }
+                DataOutput::Multi {
+                    output_writer,
+                    shuffle_block_writer,
+                    spill,
+                    spill_reader,
+                    recycled_buffer,
+                    ..
+                } => {
+                    self.offsets[pid] = output_writer.stream_position()?;
+
+                    let mut flush_timer = metrics.write_time.timer();
+                    spill.flush()?;
+                    flush_timer.stop();
+                    let ranges = spill.ranges(pid)?;
+                    if !ranges.is_empty() {
+                        if spill_reader.is_none() {
+                            let path = spill.path()?.ok_or_else(|| {
+                                DataFusionError::Internal(
+                                    "shuffle spill ranges recorded without a spill file"
+                                        .to_string(),
+                                )
+                            })?;
+                            // Open first, so a failed open leaves nothing charged.
+                            let spill_file = File::open(path)?;
+                            let buffer_size =
+                                self.buffers.as_ref().map_or(write_buffer_size, |buffers| {
+                                    super::reserve_buffer(buffers, write_buffer_size)

Review Comment:
   The first `finish_partition` now flushes the spill file and releases its write buffer before anything else, then sizes the copy buffer and only then the data file buffer. The copy buffer serves every spilled range; the data file buffer matters less, because blocks of the write buffer size bypass it, so it is the one that falls back to 8 KiB when the pool is short. `output_buffers_are_sized_after_the_spill_buffer_is_released` checks the capacities at pools that fit two full buffers, one, and none, with identical output, and the last case still copies a large range through `io::copy`. `failed_first_partition_keeps_the_reservation_in_sync` fails a partition after the release and checks that the reservation still equals what the writer holds.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: charge the local shuffle writer's buffers to the memory pool [datafusion-comet]

Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #6336:
URL: https://github.com/apache/datafusion-comet/pull/6336#discussion_r4380930872


##########
native/shuffle/src/writers/local/local_partition_writer.rs:
##########
@@ -372,14 +452,45 @@ impl PartitionWriter for LocalPartitionWriter {
 
         // The shuffle output is complete; nothing else encodes through this context.
         self.codec_context.release_zstd();
+        self.sync_buffer_reservation();
 
         Ok(())
     }
 
     fn write_burst_complete(&mut self) {
-        // A spill burst just ended and the next encode may be a long time coming; the zstd
-        // workspace is native memory no reservation tracks, so don't sit on it.
+        // A spill burst just ended and the next encode may be a long time coming, so don't
+        // sit on the zstd workspace.
         self.codec_context.release_zstd();
+        self.sync_buffer_reservation();
+        // A pressure spill opened its file while the batches still held the pool; now they don't.
+        if let (Some(buffers), DataOutput::Multi { spill, .. }) =
+            (&self.buffers, &mut self.data_output)
+        {
+            spill.restore_buffer(buffers);
+        }
+    }
+
+    fn attach_buffer_reservation(
+        &mut self,
+        reservation: MemoryReservation,
+    ) -> datafusion::common::Result<()> {
+        let DataOutput::Multi { output_writer, .. } = &mut self.data_output else {
+            return Ok(());
+        };
+        // Attached once, before any write: the rebuild below must not discard buffered bytes.
+        debug_assert!(self.buffers.is_none() && output_writer.buffer().is_empty());
+        // The data file buffer was allocated before the consumer existed; shrink it if refused.
+        let capacity = super::reserve_buffer(&reservation, self.write_buffer_size);
+        if capacity != output_writer.capacity() {
+            let output_file = output_writer
+                .get_ref()
+                .try_clone()

Review Comment:
   The `try_clone()` is gone. The data file writer starts unbuffered around the original `File`, and the first `finish_partition` moves that `File` into a writer at the granted capacity, so no second descriptor is opened.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: charge the local shuffle writer's buffers to the memory pool [datafusion-comet]

Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #6336:
URL: https://github.com/apache/datafusion-comet/pull/6336#issuecomment-6073924290

   > Could `spill()` call `write_burst_complete` after `self.reservation.free()`, and have the local writer retry the full-size spill buffer there?
   
   Done in adb9daa02. `spill()` now calls `write_burst_complete` after `reservation.free()`, and `PartitionedSpill::restore_buffer` grows the reservation by the difference with `try_grow`, takes the writer back with `BufWriter::into_parts`, and wraps it in a full-size `BufWriter` that takes over the buffered bytes. It moves those bytes instead of flushing them: they always fit the larger empty buffer, so the swap writes nothing and cannot fail, which matters because `write_burst_complete` returns nothing. A refusal leaves the 8 KiB buffer in place and the next spill asks again. A spill file that already failed a write is not grown.
   
   The data file buffer is sized when the writer attaches, before the consumer holds anything, so the writer's own batches don't shrink it. The copy buffer is sized during the final write while the last batches are still charged and is not grown back; a range larger than it goes through `io::copy`.
   
   `pressure_spill_restores_the_spill_buffer` spills with no buffer limit in 1, 4 and 16 MiB pools and checks that the spill buffer is back to full size after every spill, that the reservation equals the bytes held, and that every row is written once.
   
   `shuffle_bench` on `lineitem` at the SF1 row count, 200 partitions, lz4, no buffer limit, `ci` profile, 7 rounds with the build order rotated, in a Linux container. Median shuffle write time:
   
   | Pool | main | before | after |
   |---|---|---|---|
   | 16 MiB | 0.128 s | 0.130 s | 0.130 s |
   | 32 MiB | 0.127 s | 0.146 s | 0.129 s |
   | 64 MiB | 0.126 s | 0.130 s | 0.120 s |
   
   On this data the 8 KiB buffer showed up at 32 MiB rather than 16.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: charge the local shuffle writer's buffers to the memory pool [datafusion-comet]

Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #6336:
URL: https://github.com/apache/datafusion-comet/pull/6336#discussion_r4380808224


##########
native/shuffle/src/writers/local/local_partition_writer.rs:
##########
@@ -372,14 +452,45 @@ impl PartitionWriter for LocalPartitionWriter {
 
         // The shuffle output is complete; nothing else encodes through this context.
         self.codec_context.release_zstd();
+        self.sync_buffer_reservation();
 
         Ok(())
     }
 
     fn write_burst_complete(&mut self) {
-        // A spill burst just ended and the next encode may be a long time coming; the zstd
-        // workspace is native memory no reservation tracks, so don't sit on it.
+        // A spill burst just ended and the next encode may be a long time coming, so don't
+        // sit on the zstd workspace.
         self.codec_context.release_zstd();
+        self.sync_buffer_reservation();
+        // A pressure spill opened its file while the batches still held the pool; now they don't.
+        if let (Some(buffers), DataOutput::Multi { spill, .. }) =
+            (&self.buffers, &mut self.data_output)
+        {
+            spill.restore_buffer(buffers);
+        }
+    }
+
+    fn attach_buffer_reservation(
+        &mut self,
+        reservation: MemoryReservation,
+    ) -> datafusion::common::Result<()> {
+        let DataOutput::Multi { output_writer, .. } = &mut self.data_output else {
+            return Ok(());
+        };
+        // Attached once, before any write: the rebuild below must not discard buffered bytes.
+        debug_assert!(self.buffers.is_none() && output_writer.buffer().is_empty());
+        // The data file buffer was allocated before the consumer existed; shrink it if refused.
+        let capacity = super::reserve_buffer(&reservation, self.write_buffer_size);

Review Comment:
   The data file writer now starts unbuffered and nothing is charged at attach. The first `finish_partition` sizes it, so an operator feeding the shuffle gets the pool until the output is written. `writer_charges_nothing_before_the_first_row` builds the repartitioner with a 1 MiB pool and the default 1 MiB write buffer, checks that nothing is reserved, has a second consumer take 128 bytes, and compares the output with an unbounded pool.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: charge the local shuffle writer's buffers to the memory pool [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #6336:
URL: https://github.com/apache/datafusion-comet/pull/6336#issuecomment-5876121871

   This is a light fully automated review since there are so many PRs open.
   
   `buffer_bytes_held` at `native/shuffle/src/writers/local/local_partition_writer.rs:195` charges `codec_context.retained_bytes()`, and that workspace is allocated by libzstd through libc `malloc` (zstd-sys only swaps the allocator on wasm). So with this PR the pool tracks C library memory that the `AccountingAllocator` behind `native_allocated` never sees, and two docs now say the opposite. The "Non-Rust allocations" bullet at `docs/source/contributor-guide/memory_management.md:406` says neither the memory pool nor the allocation counters see such memory. Step 3 of "Sizing the Overhead from the Memory Usage Log" at `docs/source/user-guide/latest/tuning/memory.md:175` says neither `allocated` nor `reserved` includes memory from native C libraries such as zstd.
   
   When `spark.comet.shuffle.compression.codec` is `zstd`, `reserved` now holds one context per multi-partition shuffle task that is mid-spill or in its final write (about 1.3 MiB at the default level and close to 8 MiB at levels 7 and 8), and that also comes straight off the `allocated - reserved` difference the tuning page sizes the overhead from. Charging it still seems right to me, since it is real memory Spark should budget for. Could those two passages call out the shuffle writer's zstd context as the exception, alongside the list this PR already updates higher up in the same tuning page?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: charge the local shuffle writer's buffers to the memory pool [datafusion-comet]

Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #6336:
URL: https://github.com/apache/datafusion-comet/pull/6336#discussion_r4380839104


##########
native/shuffle/src/writers/local/local_partition_writer.rs:
##########
@@ -372,14 +452,45 @@ impl PartitionWriter for LocalPartitionWriter {
 
         // The shuffle output is complete; nothing else encodes through this context.
         self.codec_context.release_zstd();
+        self.sync_buffer_reservation();
 
         Ok(())
     }
 
     fn write_burst_complete(&mut self) {
-        // A spill burst just ended and the next encode may be a long time coming; the zstd
-        // workspace is native memory no reservation tracks, so don't sit on it.
+        // A spill burst just ended and the next encode may be a long time coming, so don't
+        // sit on the zstd workspace.
         self.codec_context.release_zstd();
+        self.sync_buffer_reservation();

Review Comment:
   `write_burst_complete` now drops the drained encode scratch and syncs the reservation before it touches the spill buffer, so one wide row no longer leaves its scratch charged. The scratch regrows on the next encode. `wide_row_scratch_is_released_after_the_spill` is your case: LZ4, a 512 KiB pool, one 768 KiB incompressible value, then 128 one-row batches. It expects at most two spills, where the head spilled 129, and every row back from the output.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: charge the local shuffle writer's buffers to the memory pool [datafusion-comet]

Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on PR #6336:
URL: https://github.com/apache/datafusion-comet/pull/6336#issuecomment-5906825387

   @sunchao CI passed on `a888fbeb6` and it has your approval. Could it go into the merge queue?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: charge the local shuffle writer's buffers to the memory pool [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #6336:
URL: https://github.com/apache/datafusion-comet/pull/6336#issuecomment-6397420238

   Thanks @dwsmith1983. The project now limits contributors who are not committers to three non-draft PRs at a time, and you had nine open, so I am moving this one to draft. It is not a judgment on the change. I picked the three closest to merging, which are #6932, #6785 and #5867, and drafted the rest.
   
   When one of those merges, please mark the next PR ready for review. The suggested order is: #6336, #5872, #7127, #7130, #5864, #5613. Drafts are still welcome to get review comments, rebases and CI runs, and you can keep pushing to this branch.
   
   I approved this on Oct 9, so it is next in line once a slot opens. sunchao's findings from Oct 24 are still open, so please look at those while it is in draft.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: charge the local shuffle writer's buffers to the memory pool [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #6336:
URL: https://github.com/apache/datafusion-comet/pull/6336#discussion_r4368030211


##########
native/shuffle/src/writers/local/local_partition_writer.rs:
##########
@@ -372,14 +452,45 @@ impl PartitionWriter for LocalPartitionWriter {
 
         // The shuffle output is complete; nothing else encodes through this context.
         self.codec_context.release_zstd();
+        self.sync_buffer_reservation();
 
         Ok(())
     }
 
     fn write_burst_complete(&mut self) {
-        // A spill burst just ended and the next encode may be a long time coming; the zstd
-        // workspace is native memory no reservation tracks, so don't sit on it.
+        // A spill burst just ended and the next encode may be a long time coming, so don't
+        // sit on the zstd workspace.
         self.codec_context.release_zstd();
+        self.sync_buffer_reservation();
+        // A pressure spill opened its file while the batches still held the pool; now they don't.
+        if let (Some(buffers), DataOutput::Multi { spill, .. }) =
+            (&self.buffers, &mut self.data_output)
+        {
+            spill.restore_buffer(buffers);
+        }
+    }
+
+    fn attach_buffer_reservation(
+        &mut self,
+        reservation: MemoryReservation,
+    ) -> datafusion::common::Result<()> {
+        let DataOutput::Multi { output_writer, .. } = &mut self.data_output else {
+            return Ok(());
+        };
+        // Attached once, before any write: the rebuild below must not discard buffered bytes.
+        debug_assert!(self.buffers.is_none() && output_writer.buffer().is_empty());
+        // The data file buffer was allocated before the consumer existed; shrink it if refused.
+        let capacity = super::reserve_buffer(&reservation, self.write_buffer_size);

Review Comment:
   [P2] The empty output writer can take the entire available pool before its input produces a row. A hash join feeding the shuffle then fails its first allocation, even though this buffer receives its first bytes during finalization. Defer allocating and charging it until the first `finish_partition`.
   
   Evidence: With two output partitions, a 16-row `HashJoinExec`, and the default 1 MiB write buffer, the base wrote all rows with a 1 MiB pool. The head denied `HashJoinInput[0]` its first 128 bytes. Reducing the pool by one byte made the head succeed because the writer took the fallback. With eight partitions and 256 batches of 1,024 integers, the same eager allocation caused 256 spills versus four on the base. Deferring it restored four spills and preserved all 262,144 rows.



##########
native/shuffle/src/writers/local/local_partition_writer.rs:
##########
@@ -372,14 +452,45 @@ impl PartitionWriter for LocalPartitionWriter {
 
         // The shuffle output is complete; nothing else encodes through this context.
         self.codec_context.release_zstd();
+        self.sync_buffer_reservation();
 
         Ok(())
     }
 
     fn write_burst_complete(&mut self) {
-        // A spill burst just ended and the next encode may be a long time coming; the zstd
-        // workspace is native memory no reservation tracks, so don't sit on it.
+        // A spill burst just ended and the next encode may be a long time coming, so don't
+        // sit on the zstd workspace.
         self.codec_context.release_zstd();
+        self.sync_buffer_reservation();

Review Comment:
   [P2] After a pressure spill, `recycled_buffer` is empty but its full capacity stays charged. One wide row can leave that charge larger than the whole pool, forcing every subsequent narrow batch to spill. Shrink or drop drained scratch that the pool cannot fund, then synchronize the reservation before restoring the spill buffer.
   
   Evidence: With LZ4, a 512 KiB pool, and one 768 KiB incompressible value, the head held 1,062,608 bytes after the first spill, including 1,046,224 bytes of empty scratch. A separate 16 KiB request failed. Adding 128 one-row batches produced 129 spills versus one on the base. Releasing only the empty scratch restored the grant and the one-spill result. All 129 rows decoded.



##########
native/shuffle/src/writers/local/local_partition_writer.rs:
##########
@@ -245,88 +313,100 @@ impl PartitionWriter for LocalPartitionWriter {
         let write_buffer_size = self.write_buffer_size;
         let batch_size = self.batch_size;
 
-        match &mut self.data_output {
-            DataOutput::Single { writer, scratch } => {
-                // Single-partition data was already streamed via `write`, starting at
-                // offset 0 (already recorded in `self.offsets[0]`). Stream any trailing
-                // batches (normally none) without flushing; the long-lived writer is
-                // flushed once in `finish_all`.
-                for batch in iter.by_ref() {
-                    let batch = batch?;
-                    writer.write(
-                        &batch,
-                        scratch,
-                        &mut self.codec_context,
-                        &metrics.encode_time,
-                        &metrics.write_time,
-                    )?;
-                }
-            }
-            DataOutput::Multi {
-                output_writer,
-                shuffle_block_writer,
-                spill,
-                spill_reader,
-                recycled_buffer,
-                ..
-            } => {
-                self.offsets[pid] = output_writer.stream_position()?;
-
-                let mut flush_timer = metrics.write_time.timer();
-                spill.flush()?;
-                flush_timer.stop();
-                let ranges = spill.ranges(pid)?;
-                if !ranges.is_empty() {
-                    if spill_reader.is_none() {
-                        let path = spill.path()?.ok_or_else(|| {
-                            DataFusionError::Internal(
-                                "shuffle spill ranges recorded without a spill file".to_string(),
-                            )
-                        })?;
-                        *spill_reader = Some((File::open(path)?, vec![0; write_buffer_size]));
-                    }
-                    let (spill_file, buffer) = spill_reader.as_mut().unwrap();
-                    let mut write_timer = metrics.write_time.timer();
-                    for range in ranges {
-                        copy_spill_range(spill_file, buffer, range, output_writer)?;
-                    }
-                    write_timer.stop();
-                }
-
-                // Write in memory batches to output data file. Each partition uses its
-                // own writer so coalescing does not cross partition boundaries, but the
-                // scratch buffer is shared so its capacity carries over to the next one.
-                let mut buf_batch_writer = BufBatchWriter::new(
-                    shuffle_block_writer,
-                    output_writer,
-                    write_buffer_size,
-                    batch_size,
-                );
-                let codec_context = &mut self.codec_context;
-                let result: datafusion::common::Result<()> = (|| {
+        // Synced on every exit, errors included, so it matches what a failed partition left.
+        let result: datafusion::common::Result<()> = (|| {
+            match &mut self.data_output {
+                DataOutput::Single { writer, scratch } => {
+                    // Single-partition data was already streamed via `write`, starting at
+                    // offset 0 (already recorded in `self.offsets[0]`). Stream any trailing
+                    // batches (normally none) without flushing; the long-lived writer is
+                    // flushed once in `finish_all`.
                     for batch in iter.by_ref() {
                         let batch = batch?;
-                        buf_batch_writer.write(
+                        writer.write(
                             &batch,
-                            recycled_buffer,
-                            codec_context,
+                            scratch,
+                            &mut self.codec_context,
                             &metrics.encode_time,
                             &metrics.write_time,
                         )?;
                     }
-                    buf_batch_writer.flush(
-                        recycled_buffer,
-                        codec_context,
-                        &metrics.encode_time,
-                        &metrics.write_time,
-                    )
-                })();
-                // An errored partition must hand back a drained buffer, or its bytes
-                // leak into the next partition's block.
-                result.inspect_err(|_| recycled_buffer.clear())?;
+                }
+                DataOutput::Multi {
+                    output_writer,
+                    shuffle_block_writer,
+                    spill,
+                    spill_reader,
+                    recycled_buffer,
+                    ..
+                } => {
+                    self.offsets[pid] = output_writer.stream_position()?;
+
+                    let mut flush_timer = metrics.write_time.timer();
+                    spill.flush()?;
+                    flush_timer.stop();
+                    let ranges = spill.ranges(pid)?;
+                    if !ranges.is_empty() {
+                        if spill_reader.is_none() {
+                            let path = spill.path()?.ok_or_else(|| {
+                                DataFusionError::Internal(
+                                    "shuffle spill ranges recorded without a spill file"
+                                        .to_string(),
+                                )
+                            })?;
+                            // Open first, so a failed open leaves nothing charged.
+                            let spill_file = File::open(path)?;
+                            let buffer_size =
+                                self.buffers.as_ref().map_or(write_buffer_size, |buffers| {
+                                    super::reserve_buffer(buffers, write_buffer_size)

Review Comment:
   [P2] During final replay, the flushed spill writer still holds its charged buffer when the reader asks for another. A refused read buffer sends medium-sized ranges through `seek` and `io::copy` even though a full-size allocation is sitting idle. Transfer the drained spill buffer to the reader, or release its capacity and charge before choosing the read-buffer size.
   
   Evidence: On Linux, replaying 2,048 identical 12,828-byte ranges with a 2 MiB + 128 KiB pool took a median 31.8 ms on the base and 64.3 ms on the head over seven interleaved rounds. The head held a 1 MiB spill-write buffer and received an 8 KiB read buffer. Reusing the existing allocation took 35.8 ms at the same budget. A full repartitioner run produced 256 spills, 1,024 ranges, and output CRC `01ffa3c0` on both revisions.



##########
native/shuffle/src/writers/local/spill.rs:
##########
@@ -214,17 +217,63 @@ impl PartitionedSpill {
     fn ensure_spill_file_created(
         &mut self,
         runtime: &RuntimeEnv,
+        buffers: Option<&MemoryReservation>,
     ) -> datafusion::common::Result<()> {
         if self.spill_file.is_none() {
             let temp_file = runtime
                 .disk_manager
                 .create_tmp_file("shuffle writer spill")?;
-            let writer = BufWriter::with_capacity(self.write_buffer_size, temp_file.open_writer()?);
+            // Open before charging, so a failed open leaves nothing on the reservation.
+            let file = temp_file.open_writer()?;
+            let capacity = buffers.map_or(self.write_buffer_size, |reservation| {
+                super::reserve_buffer(reservation, self.write_buffer_size)
+            });
+            let writer = BufWriter::with_capacity(capacity, file);
             self.spill_file = Some(ActiveSpillFile { temp_file, writer });
         }
         Ok(())
     }
 
+    /// Grows a fallback-sized buffer back to the configured size if `buffers` grants it.
+    /// Buffered bytes move to the new buffer; nothing is written to the file.
+    pub(crate) fn restore_buffer(&mut self, buffers: &MemoryReservation) {
+        if self.failed {
+            return;
+        }
+        let Some(spill_file) = self.spill_file.take() else {
+            return;
+        };
+        let capacity = spill_file.writer.capacity();
+        if capacity >= self.write_buffer_size
+            || buffers.try_grow(self.write_buffer_size - capacity).is_err()

Review Comment:
   [P2] A successful full-size restoration can leave too little memory for even the next input batch. The writer then spills on every subsequent batch while retaining the enlarged buffer. Account for the input headroom needed after a pressure spill when choosing the restore size. An intermediate capacity can retain buffered I/O without forcing immediate re-spills.
   
   Evidence: With eight hash partitions, LZ4, `maxBufferBytes=0`, and 256 batches of 1,024 integers, raising the head's pool from 2,096 to 2,112 KiB increased spills from four to 198. The restored 1 MiB spill buffer left just 4,066 bytes free. The base spilled twice at 2,112 KiB. Capping only restoration at 256 KiB produced six spills and reduced median time from 284 to 177 ms over five interleaved runs. All rows decoded. The same 198-to-six difference persisted with the data-output buffer deferred at a 1,088 KiB pool.



##########
docs/source/user-guide/latest/tuning/memory.md:
##########
@@ -182,11 +187,16 @@ alongside the JVM's own non-heap memory. To size the overhead from it:
    non-heap memory, and add the most untracked memory seen on any executor.
 3. Add a margin on top. The log can miss the true peak between samples, and none of the figures
    includes the allocator's fragmentation and retained pages, or memory allocated by native C
-   libraries such as zstd. With native Arrow UDFs, budget the embedded Python interpreter and
-   PyArrow allocations in addition to the untracked memory calculated from the log. A PyArrow
-   result buffer later imported into the JVM can appear in the `imported from native` figure that
-   the estimate subtracts, even though Rust's `allocated` does not count it. Use the executor's
-   peak resident memory to size this additional overhead.
+   libraries such as zstd. The zstd context of the native shuffle writer is the exception: when
+   `spark.comet.shuffle.compression.codec` is `zstd`, a shuffle task writing to more than one local
+   partition reserves its context while spilling or writing its output, so that memory is in

Review Comment:
   [P2] The sizing guide says the local zstd workspace appears in `reserved` while writing. When a context exceeds the 8 MiB retention cap, `release_zstd_if_oversized()` drops it before the writer samples it. That peak appears in neither `reserved` nor Rust's `allocated` figure and can reach tens of MiB per concurrent task. Distinguish retained contexts from transient compression workspace here and in the contributor guide.
   
   Evidence: The same batch round-tripped at zstd levels 1, 9, and 19. At level 1 the context measured 1,369,625 bytes both after compression and at the accounting boundary. Levels 9 and 19 measured 15,459,865 and 93,848,215 bytes after compression, respectively, and zero at the accounting boundary.



##########
native/shuffle/src/writers/local/local_partition_writer.rs:
##########
@@ -372,14 +452,45 @@ impl PartitionWriter for LocalPartitionWriter {
 
         // The shuffle output is complete; nothing else encodes through this context.
         self.codec_context.release_zstd();
+        self.sync_buffer_reservation();
 
         Ok(())
     }
 
     fn write_burst_complete(&mut self) {
-        // A spill burst just ended and the next encode may be a long time coming; the zstd
-        // workspace is native memory no reservation tracks, so don't sit on it.
+        // A spill burst just ended and the next encode may be a long time coming, so don't
+        // sit on the zstd workspace.
         self.codec_context.release_zstd();
+        self.sync_buffer_reservation();
+        // A pressure spill opened its file while the batches still held the pool; now they don't.
+        if let (Some(buffers), DataOutput::Multi { spill, .. }) =
+            (&self.buffers, &mut self.data_output)
+        {
+            spill.restore_buffer(buffers);
+        }
+    }
+
+    fn attach_buffer_reservation(
+        &mut self,
+        reservation: MemoryReservation,
+    ) -> datafusion::common::Result<()> {
+        let DataOutput::Multi { output_writer, .. } = &mut self.data_output else {
+            return Ok(());
+        };
+        // Attached once, before any write: the rebuild below must not discard buffered bytes.
+        debug_assert!(self.buffers.is_none() && output_writer.buffer().is_empty());
+        // The data file buffer was allocated before the consumer existed; shrink it if refused.
+        let capacity = super::reserve_buffer(&reservation, self.write_buffer_size);
+        if capacity != output_writer.capacity() {
+            let output_file = output_writer
+                .get_ref()
+                .try_clone()

Review Comment:
   [P2] `try_clone()` requires a spare file descriptor solely to resize an empty `BufWriter`. If opening the data file used the executor's last descriptor and the pool refuses the configured buffer, attachment now fails with `EMFILE` before processing a row. Move the original `File` into the resized writer or wrap it once at the granted capacity.
   
   Evidence: A child process with `RLIMIT_NOFILE=32` left one descriptor for the output file. With a 64 KiB pool, a 1 MiB write buffer, and a two-partition, 16-row input requiring no spill, the base published `[0, 262, 536]`. The head opened the file, failed in `try_clone()` with OS error 24, and published no offsets.



##########
native/shuffle/src/writers/local/mod.rs:
##########
@@ -17,3 +17,18 @@
 
 pub(crate) mod local_partition_writer;
 mod spill;
+
+use datafusion::execution::memory_pool::MemoryReservation;
+
+const FALLBACK_BUFFER_SIZE: usize = 8 * 1024;
+
+/// Returns the capacity to allocate. A refused size falls back to a small buffer charged with
+/// `grow`, so the task carries on and the reservation still matches what is held.
+fn reserve_buffer(reservation: &MemoryReservation, size: usize) -> usize {
+    if reservation.try_grow(size).is_ok() {

Review Comment:
   [P2] `try_grow(size).is_ok()` treats a Spark exception as permission to use the 8 KiB fallback. If a peer's `spill()` throws `IOException`, the map can publish offsets while Spark still holds a partial grant. `restore_buffer` discards the same error, and `sync_buffer_reservation` can hide it through infallible `resize` even after the file-buffer request succeeds. Use fallible growth for reusable capacity, recover only from `ResourcesExhausted`, and propagate other errors through attachment, writing, and the post-spill hook.
   
   Evidence: With Spark 4.1.3, a 2 MiB off-heap pool, and a peer holding 1.5 MiB that threw on its first spill, both `fair_unified` and `greedy_unified` published `[0, 262, 536]` and left 524,288 bytes charged until task cleanup. The base made no peer-spill call and left zero. In a separate run the 1 MiB file grant and 256-byte input grant both succeeded. Spark threw on the 516-byte scratch charge, yet the head finished and stranded 256 bytes. Injecting `External` only during spill-buffer restoration also allowed publication.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: charge the local shuffle writer's buffers to the memory pool [datafusion-comet]

Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on code in PR #6336:
URL: https://github.com/apache/datafusion-comet/pull/6336#discussion_r4380962173


##########
docs/source/user-guide/latest/tuning/memory.md:
##########
@@ -182,11 +187,16 @@ alongside the JVM's own non-heap memory. To size the overhead from it:
    non-heap memory, and add the most untracked memory seen on any executor.
 3. Add a margin on top. The log can miss the true peak between samples, and none of the figures
    includes the allocator's fragmentation and retained pages, or memory allocated by native C
-   libraries such as zstd. With native Arrow UDFs, budget the embedded Python interpreter and
-   PyArrow allocations in addition to the untracked memory calculated from the log. A PyArrow
-   result buffer later imported into the JVM can appear in the `imported from native` figure that
-   the estimate subtracts, even though Rust's `allocated` does not count it. Use the executor's
-   peak resident memory to size this additional overhead.
+   libraries such as zstd. The zstd context of the native shuffle writer is the exception: when
+   `spark.comet.shuffle.compression.codec` is `zstd`, a shuffle task writing to more than one local
+   partition reserves its context while spilling or writing its output, so that memory is in

Review Comment:
   The sizing guide and the contributor guide now say the context is reserved only up to level 8. Above that it exceeds the 8 MiB retention cap, `release_zstd_if_oversized` frees it after each block, and it appears in neither `reserved` nor `allocated`, so the guide asks to budget one per task writing shuffle output, about 15 MiB at level 9 and 90 MiB at level 19. The stale comment next to `release_zstd_if_oversized` in `shuffle_block_writer.rs` is fixed too.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org