You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by ke...@apache.org on 2024/07/01 07:49:06 UTC
(incubator-gluten) branch main updated: [CELEBORN][DOC] Fix Celeborn support of get-started document (#6282)
This is an automated email from the ASF dual-hosted git repository.
kerwinzhang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git
The following commit(s) were added to refs/heads/main by this push:
new c5ce55228 [CELEBORN][DOC] Fix Celeborn support of get-started document (#6282)
c5ce55228 is described below
commit c5ce552280976eba884bcae36bb02b8afc355017
Author: Nicholas Jiang <pr...@163.com>
AuthorDate: Mon Jul 1 14:49:03 2024 +0700
[CELEBORN][DOC] Fix Celeborn support of get-started document (#6282)
---
docs/get-started/ClickHouse.md | 33 +++++++++++----------------------
docs/get-started/Velox.md | 4 ++--
docs/get-started/build-guide.md | 23 ++++++++++++-----------
3 files changed, 25 insertions(+), 35 deletions(-)
diff --git a/docs/get-started/ClickHouse.md b/docs/get-started/ClickHouse.md
index ab24de7a4..38ce048fe 100644
--- a/docs/get-started/ClickHouse.md
+++ b/docs/get-started/ClickHouse.md
@@ -629,19 +629,26 @@ public read-only account：gluten/hN2xX3uQ4m
### Celeborn support
-Gluten with clickhouse backend has not yet supportted [Celeborn](https://github.com/apache/celeborn) natively as remote shuffle service using columar shuffle. However, you can still use Celeborn with row shuffle, which means a ColumarBatch will be converted to a row during shuffle.
-Below introduction is used to enable this feature:
+Gluten with clickhouse backend supports [Celeborn](https://github.com/apache/celeborn) as remote shuffle service. Currently, the supported Celeborn versions are `0.3.x` and `0.4.0`.
+
+Below introduction is used to enable this feature.
First refer to this URL(https://github.com/apache/celeborn) to setup a celeborn cluster.
+When compiling the Gluten Java module, it's required to enable `celeborn` profile, as follows:
+
+```
+mvn clean package -Pbackends-clickhouse -Pspark-3.3 -Pceleborn -DskipTests
+```
+
Then add the Spark Celeborn Client packages to your Spark application's classpath(usually add them into `$SPARK_HOME/jars`).
- Celeborn: celeborn-client-spark-3-shaded_2.12-[celebornVersion].jar
-Currently to use Celeborn following configurations are required in `spark-defaults.conf`
+Currently to use Gluten following configurations are required in `spark-defaults.conf`
```
-spark.shuffle.manager org.apache.spark.shuffle.celeborn.SparkShuffleManager
+spark.shuffle.manager org.apache.spark.shuffle.gluten.celeborn.CelebornShuffleManager
# celeborn master
spark.celeborn.master.endpoints clb-master:9097
@@ -670,24 +677,6 @@ spark.celeborn.storage.hdfs.dir hdfs://<namenode>/celeborn
spark.dynamicAllocation.enabled false
```
-#### Celeborn Columnar Shuffle Support
-Currently, the supported Celeborn versions are `0.3.x` and `0.4.0`.
-The native Celeborn support can be enabled by the following configuration
-```
-spark.shuffle.manager=org.apache.spark.shuffle.gluten.celeborn.CelebornShuffleManager
-```
-
-quickly start a celeborn cluster
-```shell
-wget https://archive.apache.org/dist/celeborn/celeborn-0.3.2-incubating/apache-celeborn-0.3.2-incubating-bin.tgz && \
-tar -zxvf apache-celeborn-0.3.2-incubating-bin.tgz && \
-mv apache-celeborn-0.3.2-incubating-bin/conf/celeborn-defaults.conf.template apache-celeborn-0.3.2-incubating-bin/conf/celeborn-defaults.conf && \
-mv apache-celeborn-0.3.2-incubating-bin/conf/log4j2.xml.template apache-celeborn-0.3.2-incubating-bin/conf/log4j2.xml && \
-mkdir /opt/hadoop && chmod 777 /opt/hadoop && \
-echo -e "celeborn.worker.flusher.threads 4\nceleborn.worker.storage.dirs /tmp\nceleborn.worker.monitor.disk.enabled false" > apache-celeborn-0.3.2-incubating-bin/conf/celeborn-defaults.conf && \
-bash apache-celeborn-0.3.2-incubating-bin/sbin/start-master.sh && bash apache-celeborn-0.3.2-incubating-bin/sbin/start-worker.sh
-```
-
### Columnar shuffle mode
We have two modes of columnar shuffle
1. prefer cache
diff --git a/docs/get-started/Velox.md b/docs/get-started/Velox.md
index d65b94fc1..5f9ae2a46 100644
--- a/docs/get-started/Velox.md
+++ b/docs/get-started/Velox.md
@@ -224,11 +224,11 @@ Currently there are several ways to asscess S3 in Spark. Please refer [Velox S3]
Gluten with velox backend supports [Celeborn](https://github.com/apache/celeborn) as remote shuffle service. Currently, the supported Celeborn versions are `0.3.x` and `0.4.0`.
-Below introduction is used to enable this feature
+Below introduction is used to enable this feature.
First refer to this URL(https://github.com/apache/celeborn) to setup a celeborn cluster.
-When compiling the Gluten Java module, it's required to enable `rss` profile, as follows:
+When compiling the Gluten Java module, it's required to enable `celeborn` profile, as follows:
```
mvn clean package -Pbackends-velox -Pspark-3.3 -Pceleborn -DskipTests
diff --git a/docs/get-started/build-guide.md b/docs/get-started/build-guide.md
index b2e4b9560..dc4989bc8 100644
--- a/docs/get-started/build-guide.md
+++ b/docs/get-started/build-guide.md
@@ -55,17 +55,18 @@ Please set them via `--`, e.g., `--velox_home=/YOUR/PATH`.
### Maven build parameters
The below parameters can be set via `-P` for mvn.
-| Parameters | Description | Default state |
-|---------------------|------------------------------------------------------------------------------|---------------|
-| backends-velox | Build Gluten Velox backend. | disabled |
-| backends-clickhouse | Build Gluten ClickHouse backend. | disabled |
-| rss | Build Gluten with Remote Shuffle Service, only applicable for Velox backend. | disabled |
-| delta | Build Gluten with Delta Lake support. | disabled |
-| iceberg | Build Gluten with Iceberg support. | disabled |
-| spark-3.2 | Build Gluten for Spark 3.2. | enabled |
-| spark-3.3 | Build Gluten for Spark 3.3. | disabled |
-| spark-3.4 | Build Gluten for Spark 3.4. | disabled |
-| spark-3.5 | Build Gluten for Spark 3.5. | disabled |
+| Parameters | Description | Default state |
+|---------------------|---------------------------------------|---------------|
+| backends-velox | Build Gluten Velox backend. | disabled |
+| backends-clickhouse | Build Gluten ClickHouse backend. | disabled |
+| celeborn | Build Gluten with Celeborn. | disabled |
+| uniffle | Build Gluten with Uniffle. | disabled |
+| delta | Build Gluten with Delta Lake support. | disabled |
+| iceberg | Build Gluten with Iceberg support. | disabled |
+| spark-3.2 | Build Gluten for Spark 3.2. | enabled |
+| spark-3.3 | Build Gluten for Spark 3.3. | disabled |
+| spark-3.4 | Build Gluten for Spark 3.4. | disabled |
+| spark-3.5 | Build Gluten for Spark 3.5. | disabled |
## Gluten Jar for Deployment
The gluten jar built out is under `GLUTEN_SRC/package/target/`.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org