[PR] chore: extract shuffle module into separate crate [datafusion-comet]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove opened a new pull request, #3749:
URL: https://github.com/apache/datafusion-comet/pull/3749

   ## Which issue does this PR close?
   
   Closes #.
   
   ## Rationale for this change
   
   Continue the crate extraction effort started in #3667 (which extracted `common` and `jni-bridge`). The shuffle module has a nearly one-directional dependency graph, making it an ideal candidate for extraction into its own crate. This improves code modularity and separation of concerns.
   
   ## What changes are included in this PR?
   
   **Move shared utilities to `common` crate:**
   - `bytes_to_i128` function moved to `common/src/utils.rs` (re-exported from core for backward compat)
   - `tracing.rs` module moved to `common/src/tracing.rs` (re-exported from core). Upgraded `once_cell::sync::Lazy` to `std::sync::LazyLock` (stable since Rust 1.80)
   
   **Create `datafusion-comet-shuffle` crate (`native/shuffle/`):**
   - All 16 shuffle source files moved from `native/core/src/execution/shuffle/`
   - Public API unchanged: `CompressionCodec`, `ShuffleBlockWriter`, `read_ipc_compressed`, `CometPartitioning`, `ShuffleWriterExec`, `spark_unsafe` module
   - `shuffle_writer` and `row_columnar` benchmarks moved to the new crate
   
   **Update core:**
   - Core re-exports the shuffle crate as `pub use datafusion_comet_shuffle as shuffle` in `execution/mod.rs`, so existing `crate::execution::shuffle::*` import paths in planner.rs, jni_api.rs, and parquet_writer.rs continue to work unchanged
   
   **Housekeeping:**
   - Added README files for `common`, `jni-bridge`, and `shuffle` crates
   - All crates have `publish = false`
   
   ## How are these changes tested?
   
   - `cargo build --workspace` passes
   - `cargo clippy --all-targets --workspace -- -D warnings` passes
   - `cargo bench -p datafusion-comet-shuffle --no-run` compiles benchmarks
   - No functional changes — pure refactoring, verified by workspace build


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] chore: extract shuffle module into separate crate [datafusion-comet]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove merged PR #3749:
URL: https://github.com/apache/datafusion-comet/pull/3749


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org