You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/08/28 21:49:18 UTC

[I] Native Celeborn shuffle: the remote read path costs up to 4.5x local decode and is unbenchmarked [datafusion-comet]

andygrove opened a new issue, #5535:
URL: https://github.com/apache/datafusion-comet/issues/5535

   Discovered during post-merge review of #5531.
   
   **Component**: `native/shuffle/src/ipc.rs` (`read_single_batch`),
   `native/core/src/execution/jni_api.rs:1265` (`decodeShuffleBlockWithValidation`)
   
   `read_ipc_compressed_validated` drops `with_skip_validation(true)`, so every remotely fetched batch
   pays full Arrow buffer and offset validation. That is the right call for untrusted bytes, but the
   cost is significant and is not recorded anywhere. Since the justification for a native Celeborn
   reader is that it beats Celeborn's own row decoder, this number should be measured before 8/n
   enables the path end to end.
   
   ## Measurement
   
   Release build, one `ShuffleBlockWriter` frame of 20 columns (10 `Int32`, 10 `Utf8`) x 8192 rows,
   300 iterations after 20 warm-up, Apple silicon. "local" is `read_ipc_compressed`; "validated" is
   `read_ipc_compressed_validated` + `validate_remote_schema`, i.e. what the reducer runs.
   
   | codec    | frame bytes | local (us/blk) | validated (us/blk) | ratio |
   | -------- | ----------- | -------------- | ------------------ | ----- |
   | None     | 1,570,252   | 38.8           | 175.2              | 4.52x |
   | Lz4Frame | 752,215     | 225.8          | 356.6              | 1.58x |
   | Snappy   | 821,025     | 558.5          | 696.5              | 1.25x |
   | Zstd(1)  | 235,724     | 636.9          | 773.1              | 1.21x |
   
   For reference, the local path itself did not regress in #5531: against the pre-PR implementation it
   measured -8.8% (None), +0.8% (Lz4), +1.1% (Snappy), -1.8% (Zstd). The table above is purely the
   price of the new remote path.
   
   ## Two hoistable costs on the same path
   
   - `decodeShuffleBlockWithValidation` calls `ShuffleScan::decode` and rebuilds `expected_types` via
     `to_arrow_datatype` on **every block**. Both are constant per reducer.
   - `env.convert_byte_array(expected_schema)` copies the serialized schema into a fresh `Vec` per
     block.
   
   Caching the decoded `Vec<DataType>` once per reducer, e.g. behind a native handle, would remove
   both.
   
   ## Ask
   
   Publish a benchmark for the remote read path in the 8/n description, and decide whether the
   validation cost is acceptable or should be made configurable.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Native Celeborn shuffle: the remote read path costs up to 4.5x local decode and is unbenchmarked [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao closed issue #5535: Native Celeborn shuffle: the remote read path costs up to 4.5x local decode and is unbenchmarked
URL: https://github.com/apache/datafusion-comet/issues/5535


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Native Celeborn shuffle: the remote read path costs up to 4.5x local decode and is unbenchmarked [datafusion-comet]

Posted by "pingzh (via GitHub)" <gi...@apache.org>.
pingzh commented on issue #5535:
URL: https://github.com/apache/datafusion-comet/issues/5535#issuecomment-5484247965

   Follow-up to #5352 (Support Apache Celeborn remote shuffle service), discovered during post-merge review of #5531.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org