You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/08/28 21:49:18 UTC
[I] Native Celeborn shuffle: the remote read path costs up to 4.5x local decode and is unbenchmarked [datafusion-comet]
andygrove opened a new issue, #5535:
URL: https://github.com/apache/datafusion-comet/issues/5535
Discovered during post-merge review of #5531.
**Component**: `native/shuffle/src/ipc.rs` (`read_single_batch`),
`native/core/src/execution/jni_api.rs:1265` (`decodeShuffleBlockWithValidation`)
`read_ipc_compressed_validated` drops `with_skip_validation(true)`, so every remotely fetched batch
pays full Arrow buffer and offset validation. That is the right call for untrusted bytes, but the
cost is significant and is not recorded anywhere. Since the justification for a native Celeborn
reader is that it beats Celeborn's own row decoder, this number should be measured before 8/n
enables the path end to end.
## Measurement
Release build, one `ShuffleBlockWriter` frame of 20 columns (10 `Int32`, 10 `Utf8`) x 8192 rows,
300 iterations after 20 warm-up, Apple silicon. "local" is `read_ipc_compressed`; "validated" is
`read_ipc_compressed_validated` + `validate_remote_schema`, i.e. what the reducer runs.
| codec | frame bytes | local (us/blk) | validated (us/blk) | ratio |
| -------- | ----------- | -------------- | ------------------ | ----- |
| None | 1,570,252 | 38.8 | 175.2 | 4.52x |
| Lz4Frame | 752,215 | 225.8 | 356.6 | 1.58x |
| Snappy | 821,025 | 558.5 | 696.5 | 1.25x |
| Zstd(1) | 235,724 | 636.9 | 773.1 | 1.21x |
For reference, the local path itself did not regress in #5531: against the pre-PR implementation it
measured -8.8% (None), +0.8% (Lz4), +1.1% (Snappy), -1.8% (Zstd). The table above is purely the
price of the new remote path.
## Two hoistable costs on the same path
- `decodeShuffleBlockWithValidation` calls `ShuffleScan::decode` and rebuilds `expected_types` via
`to_arrow_datatype` on **every block**. Both are constant per reducer.
- `env.convert_byte_array(expected_schema)` copies the serialized schema into a fresh `Vec` per
block.
Caching the decoded `Vec<DataType>` once per reducer, e.g. behind a native handle, would remove
both.
## Ask
Publish a benchmark for the remote read path in the 8/n description, and decide whether the
validation cost is acceptable or should be made configurable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Native Celeborn shuffle: the remote read path costs up to 4.5x local decode and is unbenchmarked [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao closed issue #5535: Native Celeborn shuffle: the remote read path costs up to 4.5x local decode and is unbenchmarked
URL: https://github.com/apache/datafusion-comet/issues/5535
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Native Celeborn shuffle: the remote read path costs up to 4.5x local decode and is unbenchmarked [datafusion-comet]
Posted by "pingzh (via GitHub)" <gi...@apache.org>.
pingzh commented on issue #5535:
URL: https://github.com/apache/datafusion-comet/issues/5535#issuecomment-5484247965
Follow-up to #5352 (Support Apache Celeborn remote shuffle service), discovered during post-merge review of #5531.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org