You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/01 04:32:00 UTC
(datafusion-comet) 01/01: docs: add a Celeborn integration guide (#6373)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-6373-62ed9d16d547fed1ac25260a1910893f076ab638
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit 9c42391cb9a6fe38d87f59992dd5fb4260483dbb
Author: Ping Zhang <pi...@gmail.com>
AuthorDate: Thu Oct 1 04:31:35 2026 +0000
docs: add a Celeborn integration guide (#6373)
* docs: add a Celeborn integration guide
* docs: make Celeborn shuffle fallback checks actionable
---
docs/source/user-guide/latest/celeborn.md | 133 +++++++++++++++++++++++
docs/source/user-guide/latest/configs.md | 22 +---
docs/source/user-guide/latest/index.rst | 5 +-
docs/source/user-guide/latest/metrics.md | 10 ++
docs/source/user-guide/latest/tuning.md | 4 +-
docs/source/user-guide/latest/tuning/celeborn.md | 100 +++++------------
docs/source/user-guide/latest/tuning/shuffle.md | 2 +-
7 files changed, 180 insertions(+), 96 deletions(-)
diff --git a/docs/source/user-guide/latest/celeborn.md b/docs/source/user-guide/latest/celeborn.md
new file mode 100644
index 0000000000..0f0762af43
--- /dev/null
+++ b/docs/source/user-guide/latest/celeborn.md
@@ -0,0 +1,133 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Using Comet with Apache Celeborn
+
+Comet accelerates query processing. [Apache Celeborn](https://celeborn.apache.org/) provides
+remote storage for shuffle data.
+
+## Support Status
+
+Comet's [native shuffle](tuning/shuffle.md#native-shuffle) through Celeborn is unavailable
+with the currently released Celeborn 0.6.x and 0.7.x clients.
+Setting `spark.comet.shuffle.mode=native` does not change that.
+
+You can still use Comet to accelerate supported scans, filters, and other query operators.
+Shuffle is handled by Celeborn's existing Spark integration.
+
+## Setup
+
+Start with a running Celeborn service and a compatible [Comet installation](installation.md).
+Celeborn is an optional application dependency that Comet does not bundle. Supply both the
+Comet JAR and a Celeborn Spark client matching the application's Spark and Scala versions on
+the startup classpaths of the driver and all executors.
+
+For Spark 3, the shaded client coordinates in Maven Central are
+`org.apache.celeborn:celeborn-client-spark-3-shaded_<scala-binary-version>:<celeborn-version>`.
+For example, `org.apache.celeborn:celeborn-client-spark-3-shaded_2.12:0.7.0` supplies
+`celeborn-client-spark-3-shaded_2.12-0.7.0.jar` for Spark 3 / Scala 2.12.
+
+The example below uses Spark 3.5, Scala 2.12, and Celeborn 0.7.0:
+
+1. Download the matching Comet JAR using the [installation guide](installation.md) and the
+ shaded Celeborn client from Maven Central.
+2. Install both JARs in `$SPARK_HOME/jars` on the driver and every executor, or include them
+ in the Spark image used by those processes. For JARs installed elsewhere, set
+ `spark.driver.extraClassPath` and `spark.executor.extraClassPath` to their paths before
+ startup. Those paths must exist on the corresponding machines.
+3. Start Spark with the settings below, replacing the Celeborn master endpoints with your
+ deployment's values.
+
+Spark 3.5 loads the shuffle manager before the executor's user-JAR classloader is initialized.
+Supplying only `--jars` or `--packages` does not ensure that the manager and Celeborn client
+are available at that point; use the startup classpaths above.
+
+This example enables Comet with Spark shuffle backed by Celeborn, as described in
+[Support Status](#support-status).
+
+```shell
+$SPARK_HOME/bin/spark-shell \
+ --conf spark.plugins=org.apache.spark.CometPlugin \
+ --conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometCelebornShuffleManager \
+ --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
+ --conf spark.celeborn.master.endpoints=celeborn-master-1:9097,celeborn-master-2:9097 \
+ --conf spark.comet.exec.enabled=true \
+ --conf spark.comet.shuffle.enabled=true \
+ --conf spark.comet.shuffle.mode=auto \
+ --conf spark.comet.explain.fallback.enabled=true \
+ --conf spark.memory.offHeap.enabled=true \
+ --conf spark.memory.offHeap.size=2g
+```
+
+Celeborn's shaded client requires the Kryo serializer; see the
+[Celeborn deployment guide](https://github.com/apache/celeborn/blob/v0.7.0/docs/deploy.md#deploy-spark-client).
+For another Spark or Scala version, select matching Comet and Celeborn artifacts rather than
+reusing the Spark 3 / Scala 2.12 example.
+
+Set the shuffle manager and Celeborn service configuration before creating the Spark context.
+Keep your deployment's existing Celeborn authentication, storage, and recovery settings.
+
+## Verifying the Shuffle Path
+
+1. Run a query containing a shuffle, then inspect its executed plan in the Spark SQL UI or
+ with `df.explain("formatted")`. With AQE, inspect the final plan after an action completes.
+ Supported operators can appear as Comet nodes, while the shuffle appears as a plain
+ `Exchange`. This is the expected result with the currently released Celeborn 0.6.x and
+ 0.7.x clients.
+2. Inspect shuffle read/write bytes, records, and time in the Spark UI. Spark's remote-read
+ byte counters alone cannot confirm that Celeborn stored the data: they also count local
+ shuffle files fetched from another executor. Celeborn's fallback policy may select local
+ Spark shuffle for some exchanges; use the check below to identify those shuffles.
+
+If an operator you expected Comet to accelerate remains on Spark, use
+`spark.comet.explain.fallback.enabled=true` to see the reasons in the driver log. See
+[Understanding Comet Plans](understanding-comet-plans.md) for details.
+
+### Checking for Local Fallback
+
+For the Spark 3.5 / Celeborn 0.7.0 setup above, enable INFO logging for
+`org.apache.spark.scheduler.DAGScheduler` on the driver before running the query, then:
+
+1. Open the query in the Spark UI's SQL tab, follow its associated jobs, and identify the
+ shuffle-writing stage you want to check.
+2. Find that stage in the driver log. For example,
+ `Submitting ShuffleMapStage 5 (MapPartitionsRDD[17] ...)` identifies its input RDD as 17.
+ Find the corresponding
+ `Registering RDD 17 (...) as input to shuffle 3` message. In this example, stage 5 writes
+ shuffle 3. Match the IDs within the same application; the messages need not be adjacent.
+3. Check Celeborn's driver logs for
+ `Fallback to vanilla Spark SortShuffleManager for shuffle: 3`.
+ This confirms that shuffle 3 selected local Spark shuffle. If fallback occurs with dynamic
+ allocation enabled and no external shuffle service, Celeborn instead logs an ERROR
+ containing `fallback to vanilla Spark SortShuffleManager for shuffle: 3`.
+
+The IDs above are examples. Use the shuffle ID you found in step 2 when checking Celeborn's
+[fallback messages](https://github.com/apache/celeborn/blob/v0.7.0/client-spark/spark-3/src/main/java/org/apache/spark/shuffle/celeborn/SparkShuffleManager.java#L222-L234).
+The absence of a fallback message does not prove that data was stored in Celeborn.
+
+## Troubleshooting
+
+| Symptom | What to check |
+| ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| Celeborn classes cannot be loaded when the application starts | Supply the matching shaded client on both driver and executor startup classpaths. Comet does not bundle it. |
+| The plan contains `Exchange` rather than `CometExchange` | This is expected with the currently released Celeborn 0.6.x and 0.7.x clients. Shuffle uses Celeborn's existing Spark integration; other supported operators can still run in Comet. |
+| Setting `spark.comet.shuffle.mode=native` does not change the plan | Native shuffle is disabled with these clients. Changing this setting does not enable it. |
+| No Comet operators appear in the plan | Check that the Comet plugin and native library loaded, that Comet execution is enabled, and that the query uses supported operators. Inspect the driver fallback explanations. |
+| Shuffle data is stored locally instead of in Celeborn | Check Celeborn's fallback policy, partition-count threshold, worker availability, and quota. An effective `spark.celeborn.client.spark.shuffle.fallback.policy=ALWAYS` selects local shuffle; `AUTO` can also select it. |
+| Changing the shuffle manager in the SQL session has no effect | Set the manager before creating the Spark context. Restart the application to change it. |
diff --git a/docs/source/user-guide/latest/configs.md b/docs/source/user-guide/latest/configs.md
index 59d5c711fa..0fb0ec56c4 100644
--- a/docs/source/user-guide/latest/configs.md
+++ b/docs/source/user-guide/latest/configs.md
@@ -45,24 +45,10 @@ These settings can be used to determine which parts of the plan are accelerated
## Shuffle Configuration Settings
-For native remote shuffle, `spark.comet.shuffle.rss.maxFrameBytes` limits one complete
-encoded frame, while `spark.comet.shuffle.rss.maxInFlightBytes` limits the memory reserved
-by map attempts sharing an executor's remote shuffle client. The reservation includes
-encoding workspace and overlapping frame copies. An ordinary uncompressed frame needs
-approximately seven times its size plus schema and transport overhead. The default 512 MiB
-reservation budget accommodates ordinary frames up to the default 64 MiB frame limit.
-Compressed frames still need workspace for their uncompressed data. Increase the reservation
-budget when larger rows or schemas need more workspace.
-
-If a row cannot fit the remote limits, Comet materializes a replacement shuffle using its
-local writer before publishing the exchange to downstream tasks. The replacement has a separate
-shuffle and scheduling identity, so late remote results or failures cannot affect local output.
-Independent exchanges can materialize concurrently, and runtime output statistics describe only
-the selected destination. Subsequent fetch failures retain Spark's normal recovery behavior.
-With dynamic allocation enabled, native Celeborn shuffle also requires either
-`spark.shuffle.service.enabled=true` or `spark.dynamicAllocation.shuffleTracking.enabled=true`
-(the Spark default) to preserve fallback files. Otherwise exchanges retain ordinary Spark/Celeborn
-shuffle, including applications that rely only on remote reliable storage or decommissioning.
+For Celeborn client compatibility and setup, see the [Celeborn guide](celeborn.md).
+The [native remote shuffle tuning reference](tuning/celeborn.md) explains how
+`spark.comet.shuffle.rss.maxFrameBytes` and `spark.comet.shuffle.rss.maxInFlightBytes`
+interact, including encoding workspace and local fallback for oversized rows.
<!--BEGIN:CONFIG_TABLE[shuffle]-->
<!--END:CONFIG_TABLE-->
diff --git a/docs/source/user-guide/latest/index.rst b/docs/source/user-guide/latest/index.rst
index 707ad83f04..152dd7612f 100644
--- a/docs/source/user-guide/latest/index.rst
+++ b/docs/source/user-guide/latest/index.rst
@@ -28,8 +28,8 @@ types, operators, and expressions that Comet supports, along with a compatibilit
known differences from Apache Spark.
Operational topics include reading and understanding Comet query plans, tuning, available metrics,
-and integration guides for Apache Iceberg and Kubernetes. Select a topic from the navigation menu
-to read more.
+and integration guides for Apache Iceberg, Apache Celeborn, and Kubernetes. Select a topic from
+the navigation menu to read more.
.. _toc.user-guide-links-$COMET_VERSION:
.. toctree::
@@ -96,6 +96,7 @@ to read more.
Iceberg Guide <iceberg>
Iceberg Writes <iceberg-writes>
+ Celeborn Guide <celeborn>
S3 Credential Providers <s3-credential-providers>
Kubernetes Guide <kubernetes>
diff --git a/docs/source/user-guide/latest/metrics.md b/docs/source/user-guide/latest/metrics.md
index 2ed4b17742..3aa6558c5b 100644
--- a/docs/source/user-guide/latest/metrics.md
+++ b/docs/source/user-guide/latest/metrics.md
@@ -165,6 +165,16 @@ contributes again. Whether input slices arrive in one batch or separate batches
the accounting for identical spill boundaries. Other operators may still own the same buffers,
so this measures memory released from shuffle buffering, not necessarily a drop in process memory.
+### Celeborn Shuffle
+
+With the currently released Celeborn 0.6.x and 0.7.x clients, shuffle uses Celeborn's existing
+Spark integration and reports its shuffle metrics in the Spark UI. Comet's operator metrics
+still apply to the other parts of the query that run in Comet.
+
+Spark's remote-read counters do not identify the storage destination: local shuffle files
+fetched from another executor also count as remote reads. See
+[Verifying the Shuffle Path](celeborn.md#verifying-the-shuffle-path) for plan and storage checks.
+
## Native Metrics
Setting `spark.comet.explain.native.enabled=true` will cause native plans to be logged in each executor. Metrics are
diff --git a/docs/source/user-guide/latest/tuning.md b/docs/source/user-guide/latest/tuning.md
index f8d1807a97..158883bb6c 100644
--- a/docs/source/user-guide/latest/tuning.md
+++ b/docs/source/user-guide/latest/tuning.md
@@ -27,8 +27,8 @@ guide is split into the following pages:
memory overhead, choosing a memory pool, batch size, and limiting spill disk usage.
- [Shuffle Tuning](tuning/shuffle.md): enabling Comet shuffle, the native and columnar shuffle
implementations, and shuffle compression.
-- [Remote Shuffle with Celeborn](tuning/celeborn.md): using Comet native shuffle with Apache
- Celeborn.
+- [Remote Shuffle with Celeborn](tuning/celeborn.md): bounding frame sizes and memory admission
+ for native remote shuffle.
- [Scan Tuning](tuning/scans.md): Parquet filter pushdown, Parquet split sizing, and Iceberg data
file concurrency.
- [Operator Tuning](tuning/operators.md): joins, adaptive partial aggregation, and sorting on
diff --git a/docs/source/user-guide/latest/tuning/celeborn.md b/docs/source/user-guide/latest/tuning/celeborn.md
index 23d3f88d84..48249ec7dd 100644
--- a/docs/source/user-guide/latest/tuning/celeborn.md
+++ b/docs/source/user-guide/latest/tuning/celeborn.md
@@ -19,76 +19,30 @@ under the License.
# Remote Shuffle with Celeborn
-Applications using Apache Celeborn can use Comet's composite shuffle manager to retain ordinary
-Spark/Celeborn shuffle while accelerating other operators with Comet.
-
-Native shuffle also requires reliable completion tracking for in-flight payloads. Released Celeborn
-0.6.0 and 0.7.0 clients do not provide the required guarantee, so these versions retain ordinary
-Spark/Celeborn shuffle even when `spark.comet.shuffle.mode=native`. Native shuffle support for
-these clients requires a safe Celeborn push-completion API. The following settings request
-native shuffle when the client passes Comet's compatibility checks:
-
-```properties
-spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometCelebornShuffleManager
-spark.comet.exec.enabled=true
-spark.comet.shuffle.enabled=true
-spark.comet.shuffle.mode=native
-spark.celeborn.client.spark.stageRerun.enabled=true
-```
-
-Set the shuffle manager and Celeborn configuration before creating the Spark context. Celeborn is
-an optional application dependency, not bundled with Comet: provide a compatible Celeborn Spark
-client matching the application's Spark and Scala versions on both the driver and executors,
-alongside the Comet JAR. Keep the application's existing Celeborn service configuration.
-
-Native Celeborn shuffle requires explicit `spark.comet.shuffle.mode=native`. The default `auto`
-mode and `jvm` mode retain ordinary Spark shuffle through the delegated Celeborn manager; they do
-not select Comet's JVM columnar shuffle. Comet execution can still accelerate other operators.
-With native mode enabled, exchanges with unsupported children, data types, or partitioning also
-retain the ordinary Spark/Celeborn shuffle path. The local `CometShuffleManager` keeps its existing
-native-to-columnar fallback behavior.
-
-Stage reruns must remain enabled so failed or ambiguous map attempts can recover through a new
-Celeborn shuffle generation. Native RSS does not support `spark.io.encryption.enabled=true`;
-encrypted applications retain ordinary Spark/Celeborn shuffle instead. Do not disable encryption
-required by the application to enable native RSS. Eligibility uses the manager's application-time
-configuration, including Celeborn's effective defaults and legacy aliases, rather than later SQL
-session overrides.
-
-Celeborn's fallback policy remains application-owned. An effective
-`spark.celeborn.client.spark.shuffle.fallback.policy=ALWAYS`, or an `AUTO` partition-count
-threshold that the exchange reaches, keeps the exchange on Spark. Worker availability and quota
-can still cause Celeborn to choose local fallback during registration. Once an exchange has been
-planned as native, Comet rejects that local handle and fails the registration: native Arrow frames
-cannot be passed to Spark's ordinary local shuffle writer. Set
-`spark.celeborn.client.spark.shuffle.fallback.policy=NEVER` only if the application also wants
-Celeborn to prohibit local fallback for ordinary Spark shuffles.
-
-Native frames retain Comet's configured compression; the raw Celeborn client path bypasses
-Celeborn's additional row compression and decompression. Use
-`spark.comet.shuffle.rss.maxFrameBytes` and `spark.comet.shuffle.rss.maxInFlightBytes` to bound
-encoded frame size and executor-side push admission. The defaults are 64 MiB and 512 MiB,
-respectively. Admission includes Arrow encoding workspace as well as overlapping native, JNI,
-and client frame copies. An uncompressed frame needs roughly seven times its size plus schema
-and codec overhead. Compression reduces the transmitted bytes but still needs uncompressed
-encoding workspace.
-
-Comet splits large batches between rows. If a single row, its schema, or its encoding workspace
-cannot fit the remote limits, Comet abandons the remote shuffle and materializes a replacement
-using its local shuffle writer before downstream tasks can consume the exchange. The replacement
-has a separate shuffle and scheduling identity, so late remote results cannot overwrite or skip
-local map output, and remote stage failures cannot abort the replacement. Independent exchanges
-can materialize concurrently; readers wait for their storage decisions before execution. Runtime
-output statistics count only the selected destination. All reads and retries for the replacement
-use local files and Spark's block transfer
-service, including normal recovery after later fetch failures. Native operators and Comet's
-Arrow shuffle format are preserved, and remote admission limits remain enforced. Once remote
-output has been published, subsequent failures use the existing Spark/Celeborn recovery path;
-Comet does not change that shuffle's destination. Local fallback uses executor disk. When `spark.dynamicAllocation.enabled=true`, native Celeborn shuffle requires
-`spark.shuffle.service.enabled=true` or `spark.dynamicAllocation.shuffleTracking.enabled=true`
-(the Spark default) so those files remain available. Applications using dynamic allocation with
-both settings disabled retain ordinary Spark/Celeborn shuffle, even if remote reliable storage or
-decommissioning enables dynamic allocation. Executor shutdown preserves fallback files for the
-external shuffle service; explicit shuffle unregister retains the normal local cleanup behavior.
-AQE reducer coalescing and mapper-range reads are supported, but Celeborn physical-skew chunk reads
-are not.
+See the [Celeborn guide](../celeborn.md) for setup and client compatibility. The settings
+below apply only to Comet's native remote shuffle, which is unavailable with the currently
+released Celeborn 0.6.x and 0.7.x clients. They do not tune Celeborn's existing Spark shuffle
+implementation.
+
+| Setting | Default | Purpose |
+| ------------------------------------------ | ------- | --------------------------------------------------------------------------------------------------- |
+| `spark.comet.shuffle.rss.maxFrameBytes` | 64 MiB | Maximum size of one complete encoded frame. The admission budget can reduce the effective limit. |
+| `spark.comet.shuffle.rss.maxInFlightBytes` | 512 MiB | Shared memory admission budget for map attempts using the same executor-side remote shuffle client. |
+
+Admission includes Arrow encoding workspace and overlapping native, JNI, and client frame
+copies. An ordinary uncompressed frame needs roughly seven times its size plus schema and
+codec overhead. The default 512 MiB budget accommodates ordinary frames up to the default
+64 MiB frame limit. Compression reduces transmitted bytes but still needs workspace for the
+uncompressed data. This budget bounds shuffle-write admission, not total executor memory.
+
+Comet splits batches between rows. If a row, schema, or encoding workspace cannot fit, Comet
+replaces the exchange with its local native shuffle writer before downstream tasks consume
+the output.
+Increase the limiting setting if larger rows need to stay on the remote path, allowing for
+encoding workspace as well as the encoded frame. Raising only `maxFrameBytes` may not help
+when `maxInFlightBytes` is the limiting budget. Larger budgets also increase potential
+executor memory use.
+
+Native frames use Comet's [shuffle compression settings](shuffle.md#shuffle-compression).
+The raw Celeborn path bypasses Celeborn's additional row compression and decompression, so
+frames are not compressed twice.
diff --git a/docs/source/user-guide/latest/tuning/shuffle.md b/docs/source/user-guide/latest/tuning/shuffle.md
index eddd1e80f1..71c9c431ab 100644
--- a/docs/source/user-guide/latest/tuning/shuffle.md
+++ b/docs/source/user-guide/latest/tuning/shuffle.md
@@ -35,7 +35,7 @@ Once it is disabled, the configured shuffle manager handles ordinary Spark shuff
without Comet's shuffle implementation.
Applications that use Apache Celeborn set a different shuffle manager. See
-[Remote Shuffle with Celeborn](celeborn.md).
+[Celeborn guide](../celeborn.md).
## Shuffle Implementations
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org