You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/04/28 17:05:17 UTC

[PR] feat(sort-shuffle): byte-copy spill files into consolidated output [datafusion-ballista]

andygrove opened a new pull request, #1615:
URL: https://github.com/apache/datafusion-ballista/pull/1615

# Which issue does this PR close?

Closes #.

# Rationale for this change

Sort-shuffle finalize previously decoded every spilled batch and re-emitted it through an IPC `FileWriter`, paying decompress + Arrow allocation + recompress for every spilled byte. This PR replaces that round-trip with a `std::io::copy` of the spill file straight into the consolidated output. On Linux this engages `copy_file_range` / `sendfile`, so spilled bytes never re-enter user space.

# What changes are included in this PR?

The on-disk format for sort-shuffle output changes:

- **Data file**: was a single IPC File with a footer of batch-block offsets. Now it is a leading schema-header IPC stream followed by per-partition byte ranges, each holding zero or more concatenated self-contained IPC streams.
- **Index file**: was little-endian i64 cumulative batch indices (despite the docstring already promising byte offsets). Now it stores actual little-endian i64 byte offsets, matching what the docstring always claimed.
- **Reader**: `stream_sort_shuffle_partition` recovers the schema from the leading header stream and uses a new bounded multi-stream reader that crosses concatenated stream EOS markers within a partition's byte range.

Hash-based shuffle is intentionally untouched. Public API of `is_sort_shuffle_output`, `get_index_path`, and `stream_sort_shuffle_partition` is unchanged, so `ShuffleReaderExec` and the executor's Arrow Flight service work without modification.

New tests cover multi-spill, in-memory-only, and empty-partition round-trips.

# Are there any user-facing changes?

No public-API changes. The sort-shuffle on-disk format changes — it is executor-internal, but in-flight files written by older binaries are not readable by this version.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] feat(sort-shuffle): byte-copy spill files and enable block-IO transport [datafusion-ballista]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.

andygrove commented on PR #1615:
URL: https://github.com/apache/datafusion-ballista/pull/1615#issuecomment-4338982157

   The performance benefits in this PR are:
   
   1. make it cheaper to combine the spill files into the final file - file copy rather than re-encode
   2. allow the use of `IO_BLOCK_TRANSPORT` with sort-based shuffle
   
   The standalone benchmark only measures the first item. I did not see a significant performance improvement with local tests. The numbers were about the same.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] feat(sort-shuffle): byte-copy spill files and enable block-IO transport [datafusion-ballista]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.

andygrove commented on PR #1615:
URL: https://github.com/apache/datafusion-ballista/pull/1615#issuecomment-4338725867

   > this is cool @andygrove i wonder do you have some numbers? i wonder if improvement is it similar to #1315 ?
   
   I'm running benchmarks now ....


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] feat(sort-shuffle): byte-copy spill files and enable block-IO transport [datafusion-ballista]

Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.

milenkovicm merged PR #1615:
URL: https://github.com/apache/datafusion-ballista/pull/1615


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] feat(sort-shuffle): byte-copy spill files and enable block-IO transport [datafusion-ballista]

Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.

milenkovicm commented on PR #1615:
URL: https://github.com/apache/datafusion-ballista/pull/1615#issuecomment-4338720218

   this is cool @andygrove i wonder do you have some numbers? i wonder if improvement is it similar to #1315 ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org