You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "pingzh (via GitHub)" <gi...@apache.org> on 2026/08/26 22:13:52 UTC
[PR] feat: add destination-aware native shuffle execution (3/n) [datafusion-comet]
pingzh opened a new pull request, #5481:
URL: https://github.com/apache/datafusion-comet/pull/5481
## Which issue does this PR close?
Part of #5352. This is the third foundational PR and does not close the issue.
Previous PRs:
- https://github.com/apache/datafusion-comet/pull/5473
- https://github.com/apache/datafusion-comet/pull/5476
## Rationale for this change
Previous PRs introduced the RSS partition writer and destination-aware shuffle plans, but native shuffle execution still supports only local files. This PR enables the existing shuffle execution path to write to either local files or a remote shuffle service.
## What changes are included in this PR?
- Introduce `ShuffleWriterDestination` with local and RSS destinations.
- Add a destination-aware `ShuffleWriterExec` constructor while preserving the existing local constructor.
- Reuse existing partitioners for both local and remote shuffle output.
- Preserve typed callback errors and enforce RSS frame-size limits.
- Update spill logging and metrics documentation for remote destinations.
- Add end-to-end coverage for compression codecs, partitioning strategies, empty schemas, remote spilling, callback failures, and local compatibility.
Binding RSS destinations to task-owned JVM callbacks will be addressed in a subsequent PR.
## How are these changes tested?
- Added 10 end-to-end RSS execution tests.
- Passed 98 Rust tests across shuffle, JNI bridge, protobuf, and planner suites.
- Passed all 31 tests in `CometNativeShuffleSuite`.
- Passed workspace-wide Clippy, Rust formatting checks, and the full Maven reactor build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add destination-aware native shuffle execution (3/n) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #5481:
URL: https://github.com/apache/datafusion-comet/pull/5481#issuecomment-5432994933
Merged, thanks @pingzh !!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add destination-aware native shuffle execution (3/n) [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao merged PR #5481:
URL: https://github.com/apache/datafusion-comet/pull/5481
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org