You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "peterxcli (via GitHub)" <gi...@apache.org> on 2026/07/25 18:18:09 UTC
[PR] perf: reuse Arrow IPC compression context across shuffle blocks [datafusion-comet]
peterxcli opened a new pull request, #5038:
URL: https://github.com/apache/datafusion-comet/pull/5038
## Which issue does this PR close?
Closes #5008.
## Rationale for this change
Avoid recreating Arrow IPC compression state for every shuffle block.
## What changes are included in this PR?
Reuse one `CompressionContext` per buffered partition writer and across batches in the JVM row conversion path. Update benchmarks and callers accordingly.
## How are these changes tested?
Existing shuffle IPC round-trip, buffered writer, and shuffle scan tests pass. The shuffle writer benchmark also compiles successfully.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse Arrow IPC compression context across shuffle blocks [datafusion-comet]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on PR #5038:
URL: https://github.com/apache/datafusion-comet/pull/5038#issuecomment-5384238942
@sunchao thanks for the review! Do you think this is ready to merge now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse Arrow IPC compression context across shuffle blocks [datafusion-comet]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on PR #5038:
URL: https://github.com/apache/datafusion-comet/pull/5038#issuecomment-5082906359
arrow 58.4:
Production path | Baseline / patch blocks | FlatBuffer allocations per block | FlatBuffer bytes per block | Context allocations
-- | -- | -- | -- | --
JVM sort → row.rs | 2,688 / 2,441 | 11 / 11 | 4,136 / 4,136 | 0 / 0
Native → BufBatchWriter | 5,857 / 4,298 | 11 / 11 | 4,136 / 4,136 | 0 / 0
arrow 59.1:
1,000 blocks | Fresh context | Reused context
-- | -- | --
Allocation calls | 212,000 | 201,011
Requested bytes | 8,508,496,000 | 8,504,364,136
---
- In Arrow 58.4, `CompressionContext` contains only an optional zstd compressor—no `FlatBufferBuilder`: [compression.rs (lines 25–34)](https://github.com/apache/arrow-rs/blob/0ff81c1215cc026a1de93ce3d2078df1ecba6f09/arrow-ipc/src/compression.rs#L25-L34)
- Comet does not enable Arrow IPC compression, so that compressor remains uninitialized: https://github.com/apache/datafusion-comet/blob/8c504872a1d215a6f454eba841969e5a3a85cd94/native/shuffle/src/writers/shuffle_block_writer.rs#L107-L110
- Arrow 58.4 independently constructs a new FlatBuffer builder for every batch: [writer.rs (lines 520–526)](https://github.com/apache/arrow-rs/blob/0ff81c1215cc026a1de93ce3d2078df1ecba6f09/arrow-ipc/src/writer.rs#L520-L526).
- The quoted issue description in https://github.com/apache/datafusion-comet/pull/5006#discussion_r3640605171 matches Arrow 59.1’s newer `IpcWriteContext`, which owns the builder: [compression.rs (lines 27–38)](https://github.com/apache/arrow-rs/blob/b1de629ae82ee417e24a413f8f6815c884ec971b/arrow-ipc/src/compression.rs#L27-L38).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse Arrow IPC compression context across shuffle blocks [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5038:
URL: https://github.com/apache/datafusion-comet/pull/5038#issuecomment-5080764360
Thanks @peterxcli. Can you share any benchmark results?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse Arrow IPC compression context across shuffle blocks [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #5038:
URL: https://github.com/apache/datafusion-comet/pull/5038#issuecomment-5387092772
Merged, thanks @peterxcli !
I think later on we also need to bump Arrow to 59.1+ in order to get the allocation savings.
In Arrow 58.4.0, its `CompressionContext` contains only an optional zstd compressor, which Comet never initializes because Arrow IPC buffer compression is disabled. The `FlatBuffer` builder is allocated separately on every batch. Moving this unused context outside the loop therefore saves no allocations today. Comet’s outer compression encoder is still recreated per block.
Starting with Arrow 59.1, the context owns the reusable `FlatBuffer` builder. At that point, this change genuinely reduces repeated metadata-allocation work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse Arrow IPC compression context across shuffle blocks [datafusion-comet]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on PR #5038:
URL: https://github.com/apache/datafusion-comet/pull/5038#issuecomment-5326467856
@andygrove I've publish the benchmark, would appreciate it if you could take a look at this and see if we can get this merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: reuse Arrow IPC compression context across shuffle blocks [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao merged PR #5038:
URL: https://github.com/apache/datafusion-comet/pull/5038
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org