You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "peterxcli (via GitHub)" <gi...@apache.org> on 2026/08/25 13:46:19 UTC

[PR] fix: release native shuffle reservation after spill failure [datafusion-comet]

peterxcli opened a new pull request, #5461:
URL: https://github.com/apache/datafusion-comet/pull/5461

   ## Which issue does this PR close?
   
   Part of #5212 (finding 13).
   
   ## Rationale for this change
   
   `MultiPartitionShuffleRepartitioner::spill` takes ownership of its buffered batches and partition indices before calling `PartitionWriter::write`. If a write failed, the early return skipped reservation release and pinned/repeated-spill cleanup even though those batches had already been dropped.
   
   The deterministic reproducer left 96 bytes reserved after the injected failure. It also retained one pinned buffer and 32 repeated-spill bytes while the buffered batches and indices were already empty.
   
   ## What changes are included in this PR?
   
   - Preserve the first partition-writer error while running the existing spill cleanup unconditionally.
   - Drop the taken partitioned batches before releasing their reservation.
   - Add a fake failing `PartitionWriter` test covering exact reservation, pinned/repeated state, and spill-metric deltas on both success and failure paths.
   
   ## How are these changes tested?
   
   - `cargo test -p datafusion-comet-shuffle --lib` (35 tests)
   - `cargo clippy -p datafusion-comet-shuffle --lib --tests -- -D warnings`
   - `cargo fmt --all -- --check`
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: release native shuffle reservation after spill failure [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #5461:
URL: https://github.com/apache/datafusion-comet/pull/5461#issuecomment-5416293453

   Merged, thanks @peterxcli !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: release native shuffle reservation after spill failure [datafusion-comet]

Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on PR #5461:
URL: https://github.com/apache/datafusion-comet/pull/5461#issuecomment-5413294381

   @ywskycn sorry I dont know that. if you have prior work, please feel free to close, happy to review then :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: release native shuffle reservation after spill failure [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao merged PR #5461:
URL: https://github.com/apache/datafusion-comet/pull/5461


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: release native shuffle reservation after spill failure [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #5461:
URL: https://github.com/apache/datafusion-comet/pull/5461#issuecomment-5412938033

   cc @ywskycn are you also working on the same thing?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org