You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by ph...@apache.org on 2026/03/30 15:51:47 UTC
(gluten) branch main updated: [DOC] Update documentation for Spark version support and TLP graduation (#11851)
This is an automated email from the ASF dual-hosted git repository.
philo pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gluten.git
The following commit(s) were added to refs/heads/main by this push:
new 91162e1079 [DOC] Update documentation for Spark version support and TLP graduation (#11851)
91162e1079 is described below
commit 91162e107981afad9e5cd25a24eb6be25166482a
Author: Ankita Victor <an...@microsoft.com>
AuthorDate: Mon Mar 30 21:21:40 2026 +0530
[DOC] Update documentation for Spark version support and TLP graduation (#11851)
---
README.md | 4 ++--
docs/developers/HowTo.md | 4 ++--
docs/developers/MicroBenchmarks.md | 4 ++--
docs/developers/NewToGluten.md | 2 +-
docs/get-started/Velox.md | 10 ++++++----
docs/get-started/build-guide.md | 6 +++++-
docs/get-started/getting-started.md | 2 +-
docs/index.md | 6 +++---
8 files changed, 22 insertions(+), 16 deletions(-)
diff --git a/README.md b/README.md
index 3071db9c48..ea54568d33 100644
--- a/README.md
+++ b/README.md
@@ -56,7 +56,7 @@ Gluten's key components:
* **Columnar Shuffle**: Handles shuffling of Gluten's columnar data. The shuffle service of Spark core is reused, while a columnar exchange operator is implemented to support Gluten's columnar data format.
* **Fallback Mechanism**: Provides fallback to vanilla Spark for unsupported operators. Gluten's ColumnarToRow (C2R) and RowToColumnar (R2C) convert data between Gluten's columnar format and Spark's internal row format to support fallback transitions.
* **Metrics**: Collected from Gluten native engine to help monitor execution, identify bugs, and diagnose performance bottlenecks. The metrics are displayed in Spark UI.
-* **Shim Layer**: Ensures compatibility with multiple Spark versions. Gluten supports the latest 3–4 Spark releases during its development cycle, and currently supports Spark 3.2, 3.3, 3.4, and 3.5.
+* **Shim Layer**: Ensures compatibility with multiple Spark versions. Gluten supports the latest 3–4 Spark releases during its development cycle, and currently supports Spark 3.3, 3.4, 3.5, 4.0, and 4.1.
## 3. User Guide
@@ -113,7 +113,7 @@ Welcome to contribute to the Gluten project! See [CONTRIBUTING.md](CONTRIBUTING.
## 6. Community
-Gluten successfully became an Apache Incubator project in March 2024. Here are several ways to connect with the community.
+Gluten successfully became an Apache Incubator project in March 2024 and graduated as an Apache Top-Level Project in March 2026. Here are several ways to connect with the community.
### GitHub
diff --git a/docs/developers/HowTo.md b/docs/developers/HowTo.md
index 256473a65c..2ac2d9f44c 100644
--- a/docs/developers/HowTo.md
+++ b/docs/developers/HowTo.md
@@ -53,7 +53,7 @@ ${GLUTEN_HOME}/dev/builddeps-veloxbe.sh --build_tests=ON --build_benchmarks=ON -
```
cd ${GLUTEN_HOME}
-mvn test -Pspark-3.2 -Pbackends-velox -pl backends-velox \
+mvn test -Pspark-3.5 -Pbackends-velox -pl backends-velox \
-am -DtagsToInclude="org.apache.gluten.tags.GenerateExample" \
-Dtest=none -DfailIfNoTests=false \
-Dexec.skip
@@ -61,7 +61,7 @@ mvn test -Pspark-3.2 -Pbackends-velox -pl backends-velox \
- After the above operations, the example files are generated under `${GLUTEN_HOME}/backends-velox`
- You can check it by the command `tree ${GLUTEN_HOME}/backends-velox/generated-native-benchmark/`
-- You may replace `-Pspark-3.2` with `-Pspark-3.3` if your spark's version is 3.3
+- You may replace `-Pspark-3.5` with `-Pspark-3.3` or `-Pspark-3.4` for earlier Spark versions
```shell
$ tree ${GLUTEN_HOME}/backends-velox/generated-native-benchmark/
diff --git a/docs/developers/MicroBenchmarks.md b/docs/developers/MicroBenchmarks.md
index 2121d2a1de..ce0996a6cc 100644
--- a/docs/developers/MicroBenchmarks.md
+++ b/docs/developers/MicroBenchmarks.md
@@ -36,8 +36,8 @@ generate example input files:
cd /path/to/gluten/
./dev/buildbundle-veloxbe.sh --build_tests=ON --build_benchmarks=ON
-# Run test to generate input data files. If you are using spark 3.3, replace -Pspark-3.2 with -Pspark-3.3.
-mvn test -Pspark-3.2 -Pbackends-velox -pl backends-velox -am \
+# Run test to generate input data files. If you are using spark 3.3, replace -Pspark-3.5 with -Pspark-3.3.
+mvn test -Pspark-3.5 -Pbackends-velox -pl backends-velox -am \
-DtagsToInclude="org.apache.gluten.tags.GenerateExample" -Dtest=none -DfailIfNoTests=false -Dexec.skip
```
diff --git a/docs/developers/NewToGluten.md b/docs/developers/NewToGluten.md
index c957785687..6c14363b2c 100644
--- a/docs/developers/NewToGluten.md
+++ b/docs/developers/NewToGluten.md
@@ -13,7 +13,7 @@ Gluten supports Ubuntu 20.04/22.04, CentOS 7/8, and MacOS.
### JDK
-Gluten supports JDK 8 for Spark 3.2, 3.3, 3.4, and 3.5. For Spark 3.3 and later versions, Gluten
+Gluten supports JDK 8 for Spark 3.3, 3.4, and 3.5. For Spark 3.3 and later versions, Gluten
also supports JDK 11 and 17.
Note: Starting with Spark 4.0, the minimum required JDK version is 17.
diff --git a/docs/get-started/Velox.md b/docs/get-started/Velox.md
index 27ff7d7456..593a4ef880 100644
--- a/docs/get-started/Velox.md
+++ b/docs/get-started/Velox.md
@@ -92,14 +92,16 @@ Currently, Gluten is using a [IBM Velox](https://github.com/IBM/velox) which is
## compile Gluten java module and create package jar
cd /path/to/gluten
-# For spark3.2.x
-mvn clean package -Pbackends-velox -Pspark-3.2 -DskipTests
# For spark3.3.x
mvn clean package -Pbackends-velox -Pspark-3.3 -DskipTests
# For spark3.4.x
mvn clean package -Pbackends-velox -Pspark-3.4 -DskipTests
-# For spark3.5.x
+# For spark3.5.x (default)
mvn clean package -Pbackends-velox -Pspark-3.5 -DskipTests
+# For spark4.0.x (requires JDK 17+ and Scala 2.13)
+mvn clean package -Pbackends-velox -Pspark-4.0 -Pjava-17 -Pscala-2.13 -DskipTests
+# For spark4.1.x (requires JDK 17+ and Scala 2.13)
+mvn clean package -Pbackends-velox -Pspark-4.1 -Pjava-17 -Pscala-2.13 -DskipTests
```
Notes： Building Velox may fail caused by OOM. You can prevent this failure by adjusting `NUM_THREADS` (e.g., `export NUM_THREADS=4`) before building Gluten/Velox. The recommended minimal memory size is 64G.
@@ -405,7 +407,7 @@ Once built successfully, hudi features will be included in gluten-velox-bundle-X
# Coverage
-Spark3.3 has 387 functions in total. ~240 are commonly used. To get the support status of all Spark built-in functions, please refer to [Velox Backend's Supported Operators & Functions](../velox-backend-support-progress.md).
+Spark3.5 has 400+ functions in total. ~240 are commonly used. To get the support status of all Spark built-in functions, please refer to [Velox Backend's Supported Operators & Functions](../velox-backend-support-progress.md).
> Velox doesn't support [ANSI mode](https://spark.apache.org/docs/latest/sql-ref-ansi-compliance.html)), so as Gluten. Once ANSI mode is enabled in Spark config, Gluten will fallback to Vanilla Spark.
diff --git a/docs/get-started/build-guide.md b/docs/get-started/build-guide.md
index 99bf656074..cdfad12f67 100644
--- a/docs/get-started/build-guide.md
+++ b/docs/get-started/build-guide.md
@@ -28,7 +28,7 @@ Please set them via `--`, e.g. `--build_type=Release`.
| build_velox_tests | Build Velox tests. | OFF |
| build_velox_benchmarks | Build Velox benchmarks (velox_tests and connectors will be disabled if ON) | OFF |
| build_arrow | Build arrow java/cpp and install the libs in local. Can turn it OFF after first build. | ON |
-| spark_version | Build for specified version of Spark(3.3, 3.4, 3.5, ALL). `ALL` means build for all versions. | ALL |
+| spark_version | Build for specified version of Spark(3.3, 3.4, 3.5, 4.0, 4.1, ALL). `ALL` means build for all versions. | ALL |
### Velox build parameters for build-velox.sh
Please set them via `--`, e.g., `--velox_home=/YOUR/PATH`.
@@ -61,6 +61,8 @@ The below parameters can be set via `-P` for mvn.
| spark-3.3 | Build Gluten for Spark 3.3. | disabled |
| spark-3.4 | Build Gluten for Spark 3.4. | disabled |
| spark-3.5 | Build Gluten for Spark 3.5. | enabled |
+| spark-4.0 | Build Gluten for Spark 4.0. Requires JDK 17+ and Scala 2.13. | disabled |
+| spark-4.1 | Build Gluten for Spark 4.1. Requires JDK 17+ and Scala 2.13. | disabled |
## Gluten Jar for Deployment
The gluten jar built out is under `GLUTEN_SRC/package/target/`.
@@ -71,3 +73,5 @@ It's name pattern is `gluten-<backend_type>-bundle-spark<spark.bundle.version>_<
| 3.3.1 | 3.3 | 2.12 |
| 3.4.4 | 3.4 | 2.12 |
| 3.5.5 | 3.5 | 2.12 |
+| 4.0.1 | 4.0 | 2.13 |
+| 4.1.1 | 4.1 | 2.13 |
diff --git a/docs/get-started/getting-started.md b/docs/get-started/getting-started.md
index ea143e9300..34ad08a78f 100644
--- a/docs/get-started/getting-started.md
+++ b/docs/get-started/getting-started.md
@@ -24,7 +24,7 @@ Gluten supports two native backends:
- **OS**: Ubuntu 20.04/22.04 or CentOS 7/8 (other Linux distros may work with static build but are not officially tested)
- **JDK**: OpenJDK 8 or 17 (Spark 4.0 requires JDK 17+)
-- **Spark**: 3.2.2, 3.3.1, 3.4.4, 3.5.5, or 4.0.0-preview
+- **Spark**: 3.3.1, 3.4.4, 3.5.5, 4.0.1, or 4.1.1
- **Scala**: 2.12 (Spark 4.0 requires Scala 2.13)
### 2. Build
diff --git a/docs/index.md b/docs/index.md
index 76b76ef6b7..e5b769d1a4 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -3,10 +3,10 @@ layout: default
title: Home
nav_order: 1
permalink: /
-description: This site serves as a collection of documentation about the Gluten, a plugin to Double SparkSQL's Performance
+description: This site serves as a collection of documentation about Gluten, a middle layer for offloading JVM-based SQL engines' execution to native engines
---
# Overview
-Gluten: Plugin to Double SparkSQL's Performance.
+Gluten: A Middle Layer for Offloading JVM-based SQL Engines' Execution to Native Engines.
# 1 Introduction
@@ -61,4 +61,4 @@ There are several key components in Gluten:
* **Columnar Shuffle**: shuffles Gluten columnar data. The shuffle service still reuses the one in Spark core. A kind of columnar exchange operator is implemented to support Gluten columnar data format.
* **Fallback Mechanism**: supports falling back to Vanilla spark for unsupported operators. Gluten ColumnarToRow (C2R) and RowToColumnar (R2C) will convert Gluten columnar data and Spark's internal row data if needed. Both C2R and R2C are implemented in native code as well
* **Metrics**: collected from Gluten native engine to help identify bugs, performance bottlenecks, etc. The metrics are displayed in Spark UI.
-* **Shim Layer**: supports multiple Spark versions. We plan to only support Spark's latest 2 or 3 releases. Currently, Spark-3.2, Spark-3.3 & Spark-3.4 (experimental) are supported.
+* **Shim Layer**: supports multiple Spark versions. We plan to only support Spark's latest 3-4 releases. Currently, Spark 3.3, 3.4, 3.5, 4.0, and 4.1 are supported.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org