You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by su...@apache.org on 2026/08/25 05:03:35 UTC
(datafusion-comet) branch main updated: chore: add md formatting to `make format` (#5460)
This is an automated email from the ASF dual-hosted git repository.
sunchao pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/main by this push:
new bd7dc601aa chore: add md formatting to `make format` (#5460)
bd7dc601aa is described below
commit bd7dc601aa9a99289530598d4778c40e6ded7772
Author: Oleks V <co...@users.noreply.github.com>
AuthorDate: Mon Aug 24 22:03:31 2026 -0700
chore: add md formatting to `make format` (#5460)
---
Makefile | 5 +++++
docs/source/contributor-guide/native_shuffle.md | 6 ------
2 files changed, 5 insertions(+), 6 deletions(-)
diff --git a/Makefile b/Makefile
index 7c218e7275..622bae635b 100644
--- a/Makefile
+++ b/Makefile
@@ -56,6 +56,11 @@ clean:
bench:
cd native && RUSTFLAGS="-Ctarget-cpu=native" cargo bench $(FEATURES_ARG) $(filter-out $@,$(MAKECMDGOALS))
format:
+ @if command -v prettier >/dev/null 2>&1; then \
+ prettier -w "**/*.md"; \
+ else \
+ echo "prettier not found, skipping markdown formatting (npm i -g prettier)"; \
+ fi
cd native && cargo fmt
./mvnw compile test-compile scalafix:scalafix -Psemanticdb $(PROFILES)
./mvnw spotless:apply $(PROFILES)
diff --git a/docs/source/contributor-guide/native_shuffle.md b/docs/source/contributor-guide/native_shuffle.md
index 1e98bc196c..d9d675e7de 100644
--- a/docs/source/contributor-guide/native_shuffle.md
+++ b/docs/source/contributor-guide/native_shuffle.md
@@ -49,7 +49,6 @@ Native shuffle (`CometExchange`) is selected when all of the following condition
columnar output. Row-based Spark operators require JVM shuffle.
3. **Supported partitioning type**: Native shuffle supports:
-
- `HashPartitioning`
- `RangePartitioning`
- `SinglePartition`
@@ -129,7 +128,6 @@ Native shuffle (`CometExchange`) is selected when all of the following condition
1. **Plan construction**: `CometNativeShuffleWriter` builds a protobuf operator tree with a
`ShuffleWriter` operator at the root and `childNativeOp` as its child. `childNativeOp` takes
one of two shapes:
-
- The child plan's `nativeOp` directly, when `CometShuffleExchangeExec`'s child is a
`CometNativeExec` subtree. The upstream operators run inside the same `CometExecIterator`
as the writer, with no JVM-to-native batch boundary between them.
@@ -142,7 +140,6 @@ Native shuffle (`CometExchange`) is selected when all of the following condition
2. **Native execution**: A single `CometExecIterator` per partition runs the unified plan.
3. **Partitioning**: `ShuffleWriterExec` receives batches and routes to the appropriate partitioner:
-
- `MultiPartitionShuffleRepartitioner`: For hash/range/round-robin partitioning
- `SinglePartitionShufflePartitioner`: For single partition (simpler path)
@@ -150,13 +147,11 @@ Native shuffle (`CometExchange`) is selected when all of the following condition
exceeds the threshold, partitions spill to temporary files.
5. **Encoding**: `ShuffleBlockWriter` encodes each partition's data as compressed Arrow IPC:
-
- Writes compression type header
- Writes field count header
- Writes compressed IPC stream
6. **Output files**: Two files are produced:
-
- **Data file**: Concatenated partition data
- **Index file**: Array of 8-byte little-endian offsets marking partition boundaries
@@ -168,7 +163,6 @@ Native shuffle (`CometExchange`) is selected when all of the following condition
1. `CometBlockStoreShuffleReader` fetches shuffle blocks via `ShuffleBlockFetcherIterator`.
2. For each block, `NativeBatchDecoderIterator`:
-
- Reads the 8-byte compressed length header
- Reads the 8-byte field count header
- Reads the compressed IPC data
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org