You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@celeborn.apache.org by fc...@apache.org on 2023/10/14 01:54:31 UTC
[incubator-celeborn] branch main updated: [CELEBORN-987][FOLLOWUP][DOC] README#Build and sbt#System Requirements should extend to Scala 2.13 and Spark 3.5
This is an automated email from the ASF dual-hosted git repository.
fchen pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-celeborn.git
The following commit(s) were added to refs/heads/main by this push:
new f61fe1755 [CELEBORN-987][FOLLOWUP][DOC] README#Build and sbt#System Requirements should extend to Scala 2.13 and Spark 3.5
f61fe1755 is described below
commit f61fe175515d4ba02dd8debdf9c356e1729b13b9
Author: SteNicholas <pr...@163.com>
AuthorDate: Sat Oct 14 09:54:22 2023 +0800
[CELEBORN-987][FOLLOWUP][DOC] README#Build and sbt#System Requirements should extend to Scala 2.13 and Spark 3.5
### What changes were proposed in this pull request?
`README#Build` and `sbt#System Requirements` extends to Scala 2.13.
### Why are the changes needed?
`README#Build` and `sbt#System Requirements`should extend to Scala 2.13 to align the SBT CI test results.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
SBT CI tests.
Closes #1987 from SteNicholas/CELEBORN-987.
Authored-by: SteNicholas <pr...@163.com>
Signed-off-by: Fu Chen <cf...@gmail.com>
---
README.md | 27 ++++++++++++++-------------
docs/developers/sbt.md | 25 +++++++++++++------------
2 files changed, 27 insertions(+), 25 deletions(-)
diff --git a/README.md b/README.md
index 8e19998be..2472ce4c1 100644
--- a/README.md
+++ b/README.md
@@ -41,8 +41,8 @@ Celeborn Worker's slot count is decided by `total usable disk size / average shu
Celeborn worker's slot count decreases when a partition is allocated and increments when a partition is freed.
## Build
-1. Celeborn supports Spark 2.4/3.0/3.1/3.2/3.3/3.4, Flink 1.14/1.15/1.17 and Hadoop MapReduce 2/3.
-2. Celeborn tested under Scala 2.11/2.12 and Java 8/11/17 environment.
+1. Celeborn supports Spark 2.4/3.0/3.1/3.2/3.3/3.4/3.5, Flink 1.14/1.15/1.17 and Hadoop MapReduce 2/3.
+2. Celeborn tested under Scala 2.11/2.12/2.13 and Java 8/11/17 environment.
Build Celeborn
```shell
@@ -53,17 +53,18 @@ package apache-celeborn-${project.version}-bin.tgz will be generated.
> **_NOTE:_** The following table indicates the compatibility of Celeborn Spark and Flink clients with different versions of Spark and Flink for various Java and Scala versions.
-| | Java 8/Scala 2.11 | Java 8/Scala 2.12 | Java 11/Scala 2.12 | Java 17/Scala 2.12 |
-|----------------|-------------------|-------------------|--------------------|--------------------|
-| Spark 2.4 | ✔ | ❌ | ❌ | ❌ |
-| Spark 3.0 | ❌ | ✔ | ✔ | ❌ |
-| Spark 3.1 | ❌ | ✔ | ✔ | ❌ |
-| Spark 3.2 | ❌ | ✔ | ✔ | ❌ |
-| Spark 3.3 | ❌ | ✔ | ✔ | ✔ |
-| Spark 3.4 | ❌ | ✔ | ✔ | ✔ |
-| Flink 1.14 | ❌ | ✔ | ✔ | ❌ |
-| Flink 1.15 | ❌ | ✔ | ✔ | ❌ |
-| Flink 1.17 | ❌ | ✔ | ✔ | ❌ |
+| | Java 8/Scala 2.11 | Java 8/Scala 2.12 | Java 11/Scala 2.12 | Java 17/Scala 2.12 | Java 8/Scala 2.13 | Java 11/Scala 2.13 | Java 17/Scala 2.13 |
+|------------|-------------------|-------------------|--------------------|--------------------|-------------------|--------------------|--------------------|
+| Spark 2.4 | ✔ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
+| Spark 3.0 | ❌ | ✔ | ✔ | ❌ | ❌ | ❌ | ❌ |
+| Spark 3.1 | ❌ | ✔ | ✔ | ❌ | ❌ | ❌ | ❌ |
+| Spark 3.2 | ❌ | ✔ | ✔ | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.3 | ❌ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ |
+| Spark 3.4 | ❌ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ |
+| Spark 3.5 | ❌ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ |
+| Flink 1.14 | ❌ | ✔ | ✔ | ❌ | ❌ | ❌ | ❌ |
+| Flink 1.15 | ❌ | ✔ | ✔ | ❌ | ❌ | ❌ | ❌ |
+| Flink 1.17 | ❌ | ✔ | ✔ | ❌ | ❌ | ❌ | ❌ |
### Package Details
Build procedure will create a compressed package.
diff --git a/docs/developers/sbt.md b/docs/developers/sbt.md
index 8daa6c9a7..c949d0db6 100644
--- a/docs/developers/sbt.md
+++ b/docs/developers/sbt.md
@@ -22,21 +22,22 @@ Starting from version 0.4.0, the Celeborn project supports building and packagin
## System Requirements
-Celeborn Service (master/worker) supports Scala 2.11/2.12 and Java 8/11/17.
+Celeborn Service (master/worker) supports Scala 2.11/2.12/2.13 and Java 8/11/17.
The following table indicates the compatibility of Celeborn Spark and Flink clients with different versions of Spark and Flink for various Java and Scala versions:
-| | Java 8/Scala 2.11 | Java 8/Scala 2.12 | Java 11/Scala 2.12 | Java 17/Scala 2.12 |
-|----------------|-------------------|-------------------|--------------------|--------------------|
-| Spark 2.4 | ✔ | ❌ | ❌ | ❌ |
-| Spark 3.0 | ❌ | ✔ | ✔ | ❌ |
-| Spark 3.1 | ❌ | ✔ | ✔ | ❌ |
-| Spark 3.2 | ❌ | ✔ | ✔ | ❌ |
-| Spark 3.3 | ❌ | ✔ | ✔ | ✔ |
-| Spark 3.4 | ❌ | ✔ | ✔ | ✔ |
-| Flink 1.14 | ❌ | ✔ | ✔ | ❌ |
-| Flink 1.15 | ❌ | ✔ | ✔ | ❌ |
-| Flink 1.17 | ❌ | ✔ | ✔ | ❌ |
+| | Java 8/Scala 2.11 | Java 8/Scala 2.12 | Java 11/Scala 2.12 | Java 17/Scala 2.12 | Java 8/Scala 2.13 | Java 11/Scala 2.13 | Java 17/Scala 2.13 |
+|------------|-------------------|-------------------|--------------------|--------------------|-------------------|--------------------|--------------------|
+| Spark 2.4 | ✔ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
+| Spark 3.0 | ❌ | ✔ | ✔ | ❌ | ❌ | ❌ | ❌ |
+| Spark 3.1 | ❌ | ✔ | ✔ | ❌ | ❌ | ❌ | ❌ |
+| Spark 3.2 | ❌ | ✔ | ✔ | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.3 | ❌ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ |
+| Spark 3.4 | ❌ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ |
+| Spark 3.5 | ❌ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ |
+| Flink 1.14 | ❌ | ✔ | ✔ | ❌ | ❌ | ❌ | ❌ |
+| Flink 1.15 | ❌ | ✔ | ✔ | ❌ | ❌ | ❌ | ❌ |
+| Flink 1.17 | ❌ | ✔ | ✔ | ❌ | ❌ | ❌ | ❌ |
## Useful SBT commands