You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "cfmcgrady (via GitHub)" <gi...@apache.org> on 2023/08/07 12:35:48 UTC
[GitHub] [incubator-celeborn] cfmcgrady opened a new pull request, #1795: [CELEBORN-877][DOC] Document on SBT
cfmcgrady opened a new pull request, #1795:
URL: https://github.com/apache/incubator-celeborn/pull/1795
<!--
Thanks for sending a pull request! Here are some tips for you:
- Make sure the PR title start w/ a JIRA ticket, e.g. '[CELEBORN-XXXX] Your PR title ...'.
- Be sure to keep the PR description updated to reflect all changes.
- Please write your PR title to summarize what this PR proposes.
- If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
As title
### Why are the changes needed?
As title
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Manual test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1795: [CELEBORN-877][DOC] Document on SBT
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1795:
URL: https://github.com/apache/incubator-celeborn/pull/1795#discussion_r1290109122
##########
docs/developers/sbt.md:
##########
@@ -0,0 +1,276 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Building via SBT
+
+Starting from version 0.4.0, the Celeborn project supports building and packaging using SBT. This article provides a detailed guide on how to build the Celeborn project using SBT.
+
+# System Requirements
+
+Celeborn Service (master/worker) supports Scala 2.11/Scala 2.12 and Java 8/11/17.
+
+The table following indicates the compatibility of Celeborn Spark and Flink clients with different versions of Spark and Flink for various Java and Scala versions:
+
+| | Java 8/Scala 2.11 | Java 8/Scala 2.12 | Java 11/Scala 2.12 | Java 17/Scala 2.12 |
+|----------------|-------------------|-------------------|--------------------|--------------------|
+| Spark 2.4 | ✔ | ❌ | ❌ | ❌ |
+| Spark 3.0 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.1 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.2 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.3 | ❌ | ✔ | ✔ | ✔ |
+| Spark 3.4 | ❌ | ✔ | ✔ | ✔ |
+| Flink 1.14 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.15 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.17 | ❌ | ✔ | ✔ | ❌ |
+
+# Useful SBT commands
+
+## Packaging the Project
+
+As an example, one can build a version of Celeborn as follows:
+
+```
+./build/sbt clean package
+```
+
+To create a Celeborn distribution like those distributed by the [Celeborn Downloads](https://celeborn.apache.org/download/) page, and that is laid out so as to be runnable, use `./build/make-distribution.sh` in the project root directory.
+
+```
+./build/make-distribution.sh --sbt
+```
+
+## Maven-Style Profile Management
+
+We have adopted the Maven-style profile management for our Client module. For example, you can enable the Spark 3.3 client module by adding `-Pspark-3.3`:
+
+```
+# ./build/sbt -Pspark-3.3 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-spark-3
+[info] celeborn-client-spark-3-shaded
+[info] celeborn-common
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-spark-common
+[info] celeborn-spark-group
+[info] celeborn-spark-it
+[info] celeborn-worker
+```
+
+To enable the Flink 1.15 client module, add `-Pflink-1.15`:
+
+```
+# ./build/sbt -Pflink-1.15 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-flink-1_15
+[info] celeborn-client-flink-1_15-shaded
+[info] celeborn-common
+[info] celeborn-flink-common
+[info] celeborn-flink-group
+[info] celeborn-flink-it
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-worker
+```
+
+By using these profiles, you can easily switch between different client modules for Spark and Flink. These profiles enable specific dependencies and configurations relevant to the chosen version. This way, you can conveniently manage and build the desired configurations of the Celeborn project.
+
+## Building Spark/Flink Assembly Client Jars
+
+For example, you can build the Spark 3.3 client assembly jar by running the following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-spark-3-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pspark-3.3` profile:
+$ ./build/sbt -Pspark-3.3 celeborn-client-spark-3-shaded/assembly
+```
+
+Similarly, you can build the Flink 1.15 client assembly jar using the following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-flink-1_15-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pflink-1.15` profile:
+$ ./build/sbt -Pflink-1.15 celeborn-client-flink-1_15-shaded/assembly
+```
+
+By executing these commands, you will create assembly jar files for the respective Spark and Flink client modules. The assembly jar bundles all the dependencies, allowing the client module to be used independently with all required dependencies included.
+
+## Building submodules individually
+
+For instance, you can build the Celeborn Client module using:
+
+```
+$ # sbt
+$ build/sbt
+> project celeborn-client
+> package
+
+$ # Or, you can build the celeborn-client module with sbt directly using:
+$ build/sbt celeborn-client/package
+```
+
+## Testing with SBT
+
+To run all tests for the Celeborn project, you can use the following command:
+
+```shell
+./build/sbt test
+```
+
+Running tests for specific versions of Spark/Flink client.
+
+For example, to run the test cases for the Spark 3.3 client, use the following command:
+
+```shell
+$ ./build/sbt -Pspark-3.3 test
+
+$ # only run spark client related modules tests
+$ ./build/sbt -Pspark-3.3 celeborn-spark-group/test
+```
+
+Similarly, to run the test cases for the Flink 1.15 client, use the following command:
+
+```shell
+$ ./build/sbt -Pflink-1.15 test
+
+$ # only run flink client related modules tests
+$ ./build/sbt -Pflink-1.15 celeborn-flink-group/test
+```
+
+## Running Individual Tests
+
+When developing locally, it’s often convenient to run a single test or a few tests, rather than running the entire test suite.
+
+The fastest way to run individual tests is to use the sbt console. It’s fastest to keep a sbt console open, and use it to re-run tests as necessary. For example, to run all of the tests in a particular project, e.g., client:
+
+```
+$ build/sbt
+> project celeborn-client
+> test
+```
+You can run a single test suite using the `testOnly` command. For example, to run the `ShuffleClientSuiteJ`:
+
+```
+> testOnly org.apache.celeborn.client.ShuffleClientSuiteJ
+```
+The `testOnly` command accepts wildcards; e.g., you can also run the `ShuffleClientSuiteJ` with:
+
+```
+> testOnly *ShuffleClientSuiteJ
+```
+Or you could run all of the tests in the `client` package:
+
+```
+> testOnly org.apache.celeborn.client.*
+```
+If you’d like to run just a single Java test in the `ShuffleClientSuiteJ`, e.g., a test that with the name `testPushData`, you run the following command in the sbt console:
+
+```
+> testOnly *ShuffleClientSuiteJ -- *ShuffleClientSuiteJ.testPushData
+```
+
+If you’d like to run just a single Scala test in the `WorkerStatusTrackerSuite`, e.g., a test that incudes "handleHeartbeatResponse" in the name, you run the following command in the sbt console:
+
+```
+> testOnly *WorkerStatusTrackerSuite -- -z handleHeartbeatResponse
+```
+
+If you’d prefer, you can run all of these commands on the command line (but this will be slower than running tests using an open console). To do this, you need to surround `testOnly` and the following arguments in quotes:
+
+```
+$ build/sbt "celeborn-client/testOnly *WorkerStatusTrackerSuite -- -z handleHeartbeatResponse"
+```
+For more about how to run individual tests with sbt, see the [sbt documentation](https://www.scala-sbt.org/1.x/docs/Testing.html) and [JUnit Interface](https://github.com/sbt/junit-interface/#junit-interface).
+
+# Accelerating SBT
+
+This section provides instructions on setting up repository proxies for a smoother SBT experience. Depending on your location and network conditions, you can choose the appropriate approach to accelerate SBT startup and enhance dependency retrieval.
+
+## Accelerating SBT Startup
+
+The SBT startup process involves fetching the SBT bootstrap jar, which is typically obtained from the Maven Central Repository (https://repo1.maven.org/maven2/). If you encounter slow access to this repository or if it's inaccessible in your network environment, you can expedite the SBT startup by configuring a custom artifact repository using the `DEFAULT_ARTIFACT_REPOSITORY` environment variable.
+
+
+```shell
+$ # The following command fetches sbt-launch-x.y.z.jar from https://maven.aliyun.com/nexus/content/groups/public/
+$ # Ensure that the URL ends with a trailing slash "/"
+$ export DEFAULT_ARTIFACT_REPOSITORY=https://maven.aliyun.com/nexus/content/groups/public/ && ./build/sbt
Review Comment:
```suggestion
$ export DEFAULT_ARTIFACT_REPOSITORY=https://maven.aliyun.com/nexus/content/groups/public/
$ ./build/sbt
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1795: [CELEBORN-877][DOC] Document on SBT
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1795:
URL: https://github.com/apache/incubator-celeborn/pull/1795#discussion_r1290126107
##########
docs/developers/sbt.md:
##########
@@ -0,0 +1,276 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Building via SBT
+
+Starting from version 0.4.0, the Celeborn project supports building and packaging using SBT. This article provides a detailed guide on how to build the Celeborn project using SBT.
+
+# System Requirements
+
+Celeborn Service (master/worker) supports Scala 2.11/Scala 2.12 and Java 8/11/17.
+
+The table following indicates the compatibility of Celeborn Spark and Flink clients with different versions of Spark and Flink for various Java and Scala versions:
+
+| | Java 8/Scala 2.11 | Java 8/Scala 2.12 | Java 11/Scala 2.12 | Java 17/Scala 2.12 |
+|----------------|-------------------|-------------------|--------------------|--------------------|
+| Spark 2.4 | ✔ | ❌ | ❌ | ❌ |
+| Spark 3.0 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.1 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.2 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.3 | ❌ | ✔ | ✔ | ✔ |
+| Spark 3.4 | ❌ | ✔ | ✔ | ✔ |
+| Flink 1.14 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.15 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.17 | ❌ | ✔ | ✔ | ❌ |
+
+# Useful SBT commands
+
+## Packaging the Project
+
+As an example, one can build a version of Celeborn as follows:
+
+```
+./build/sbt clean package
+```
+
+To create a Celeborn distribution like those distributed by the [Celeborn Downloads](https://celeborn.apache.org/download/) page, and that is laid out so as to be runnable, use `./build/make-distribution.sh` in the project root directory.
+
+```
+./build/make-distribution.sh --sbt
+```
+
+## Maven-Style Profile Management
+
+We have adopted the Maven-style profile management for our Client module. For example, you can enable the Spark 3.3 client module by adding `-Pspark-3.3`:
+
+```
+# ./build/sbt -Pspark-3.3 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-spark-3
+[info] celeborn-client-spark-3-shaded
+[info] celeborn-common
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-spark-common
+[info] celeborn-spark-group
+[info] celeborn-spark-it
+[info] celeborn-worker
+```
+
+To enable the Flink 1.15 client module, add `-Pflink-1.15`:
+
+```
+# ./build/sbt -Pflink-1.15 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-flink-1_15
+[info] celeborn-client-flink-1_15-shaded
+[info] celeborn-common
+[info] celeborn-flink-common
+[info] celeborn-flink-group
+[info] celeborn-flink-it
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-worker
+```
+
+By using these profiles, you can easily switch between different client modules for Spark and Flink. These profiles enable specific dependencies and configurations relevant to the chosen version. This way, you can conveniently manage and build the desired configurations of the Celeborn project.
+
+## Building Spark/Flink Assembly Client Jars
+
+For example, you can build the Spark 3.3 client assembly jar by running the following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-spark-3-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pspark-3.3` profile:
+$ ./build/sbt -Pspark-3.3 celeborn-client-spark-3-shaded/assembly
+```
+
+Similarly, you can build the Flink 1.15 client assembly jar using the following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-flink-1_15-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pflink-1.15` profile:
+$ ./build/sbt -Pflink-1.15 celeborn-client-flink-1_15-shaded/assembly
+```
+
+By executing these commands, you will create assembly jar files for the respective Spark and Flink client modules. The assembly jar bundles all the dependencies, allowing the client module to be used independently with all required dependencies included.
+
+## Building submodules individually
+
+For instance, you can build the Celeborn Client module using:
+
+```
+$ # sbt
+$ build/sbt
+> project celeborn-client
+> package
+
+$ # Or, you can build the celeborn-client module with sbt directly using:
+$ build/sbt celeborn-client/package
+```
+
+## Testing with SBT
+
+To run all tests for the Celeborn project, you can use the following command:
+
+```shell
+./build/sbt test
+```
+
+Running tests for specific versions of Spark/Flink client.
+
+For example, to run the test cases for the Spark 3.3 client, use the following command:
+
+```shell
+$ ./build/sbt -Pspark-3.3 test
+
+$ # only run spark client related modules tests
+$ ./build/sbt -Pspark-3.3 celeborn-spark-group/test
+```
+
+Similarly, to run the test cases for the Flink 1.15 client, use the following command:
+
+```shell
+$ ./build/sbt -Pflink-1.15 test
+
+$ # only run flink client related modules tests
+$ ./build/sbt -Pflink-1.15 celeborn-flink-group/test
+```
+
+## Running Individual Tests
+
+When developing locally, it’s often convenient to run a single test or a few tests, rather than running the entire test suite.
+
+The fastest way to run individual tests is to use the sbt console. It’s fastest to keep a sbt console open, and use it to re-run tests as necessary. For example, to run all of the tests in a particular project, e.g., client:
+
+```
+$ build/sbt
+> project celeborn-client
+> test
+```
+You can run a single test suite using the `testOnly` command. For example, to run the `ShuffleClientSuiteJ`:
+
+```
+> testOnly org.apache.celeborn.client.ShuffleClientSuiteJ
+```
+The `testOnly` command accepts wildcards; e.g., you can also run the `ShuffleClientSuiteJ` with:
+
+```
+> testOnly *ShuffleClientSuiteJ
+```
+Or you could run all of the tests in the `client` package:
+
+```
+> testOnly org.apache.celeborn.client.*
+```
+If you’d like to run just a single Java test in the `ShuffleClientSuiteJ`, e.g., a test that with the name `testPushData`, you run the following command in the sbt console:
+
+```
+> testOnly *ShuffleClientSuiteJ -- *ShuffleClientSuiteJ.testPushData
+```
+
+If you’d like to run just a single Scala test in the `WorkerStatusTrackerSuite`, e.g., a test that incudes "handleHeartbeatResponse" in the name, you run the following command in the sbt console:
+
+```
+> testOnly *WorkerStatusTrackerSuite -- -z handleHeartbeatResponse
+```
+
+If you’d prefer, you can run all of these commands on the command line (but this will be slower than running tests using an open console). To do this, you need to surround `testOnly` and the following arguments in quotes:
+
+```
+$ build/sbt "celeborn-client/testOnly *WorkerStatusTrackerSuite -- -z handleHeartbeatResponse"
+```
+For more about how to run individual tests with sbt, see the [sbt documentation](https://www.scala-sbt.org/1.x/docs/Testing.html) and [JUnit Interface](https://github.com/sbt/junit-interface/#junit-interface).
+
+# Accelerating SBT
+
+This section provides instructions on setting up repository proxies for a smoother SBT experience. Depending on your location and network conditions, you can choose the appropriate approach to accelerate SBT startup and enhance dependency retrieval.
+
+## Accelerating SBT Startup
+
+The SBT startup process involves fetching the SBT bootstrap jar, which is typically obtained from the Maven Central Repository (https://repo1.maven.org/maven2/). If you encounter slow access to this repository or if it's inaccessible in your network environment, you can expedite the SBT startup by configuring a custom artifact repository using the `DEFAULT_ARTIFACT_REPOSITORY` environment variable.
+
+
+```shell
+$ # The following command fetches sbt-launch-x.y.z.jar from https://maven.aliyun.com/nexus/content/groups/public/
+$ # Ensure that the URL ends with a trailing slash "/"
+$ export DEFAULT_ARTIFACT_REPOSITORY=https://maven.aliyun.com/nexus/content/groups/public/ && ./build/sbt
+```
+
+This will initiate SBT using the specified repository, allowing for faster download and startup times.
+
+## Custom SBT Repositories
+
+The current repositories embedded within the Celeborn project are detailed below:
+
+```
+[repositories]
+ local
+ mavenLocal: file://${user.home}/.m2/repository/
+ local-preloaded-ivy: file:///${sbt.preloaded-${sbt.global.base-${user.home}/.sbt}/preloaded/}, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext]
+ local-preloaded: file:///${sbt.preloaded-${sbt.global.base-${user.home}/.sbt}/preloaded/}
+ # The system property value of `celeborn.sbt.default.artifact.repository` is
+ # fetched from the environment variable `DEFAULT_ARTIFACT_REPOSITORY` and
+ # assigned within the build/sbt-launch-lib.bash script.
+ private: ${celeborn.sbt.default.artifact.repository-file:///dev/null}
+ gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/repos/central/data/
+ maven-central
+ typesafe-ivy-releases: https://repo.typesafe.com/typesafe/ivy-releases/, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext], bootOnly
+ sbt-ivy-snapshots: https://repo.scala-sbt.org/scalasbt/ivy-snapshots/, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext], bootOnly
+ sbt-plugin-releases: https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
+ bintray-typesafe-sbt-plugin-releases: https://dl.bintray.com/typesafe/sbt-plugins/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
+ bintray-spark-packages: https://dl.bintray.com/spark-packages/maven/
+ typesafe-releases: https://repo.typesafe.com/typesafe/releases/
+```
+
+For numerous developers across various regions, the default repository download speeds are less than optimal. Consider, for example, developers situated in mainland China. In such cases, the following command can be employed to enhance dependency download speeds:
+
+```
+cp build/sbt-config/repositories-cn.template build/sbt-config/repositories-local
+```
+
+> NOTE: <br>
Review Comment:
is br necessary?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1795: [CELEBORN-877][DOC] Document on SBT
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1795:
URL: https://github.com/apache/incubator-celeborn/pull/1795#discussion_r1287696112
##########
mkdocs.yml:
##########
@@ -77,6 +77,7 @@ nav:
- Overview: deploy.md
- Kubernetes: deploy_on_k8s.md
- Monitoring: monitoring.md
+ - SBT Build: sbt.md
Review Comment:
Emm... I think it's kind of a developer's stuff
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1795: [CELEBORN-877][DOC] Document on SBT
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1795:
URL: https://github.com/apache/incubator-celeborn/pull/1795#discussion_r1290859913
##########
docs/developers/sbt.md:
##########
@@ -0,0 +1,281 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Building via SBT
+
+Starting from version 0.4.0, the Celeborn project supports building and packaging using SBT. This article provides a detailed guide on how to build the Celeborn project using SBT.
+
+# System Requirements
+
+Celeborn Service (master/worker) supports Scala 2.11/2.12 and Java 8/11/17.
+
+The following table indicates the compatibility of Celeborn Spark and Flink clients with different versions of Spark and Flink for various Java and Scala versions:
+
+| | Java 8/Scala 2.11 | Java 8/Scala 2.12 | Java 11/Scala 2.12 | Java 17/Scala 2.12 |
+|----------------|-------------------|-------------------|--------------------|--------------------|
+| Spark 2.4 | ✔ | ❌ | ❌ | ❌ |
+| Spark 3.0 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.1 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.2 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.3 | ❌ | ✔ | ✔ | ✔ |
+| Spark 3.4 | ❌ | ✔ | ✔ | ✔ |
+| Flink 1.14 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.15 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.17 | ❌ | ✔ | ✔ | ❌ |
+
+# Useful SBT commands
+
+## Packaging the Project
+
+As an example, one can build a version of Celeborn as follows:
+
+```
+./build/sbt clean package
+```
+
+To create a Celeborn distribution like those distributed by the [Celeborn Downloads](https://celeborn.apache.org/download/) page, and that is laid out so as to be runnable, use `./build/make-distribution.sh` in the project root directory.
+
+```
+./build/make-distribution.sh --sbt-enabled
+```
+
+## Maven-Style Profile Management
+
+We have adopted the Maven-style profile management for our Client module. For example, you can enable the Spark 3.3 client module by adding `-Pspark-3.3`:
+
+```
+# ./build/sbt -Pspark-3.3 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-spark-3
+[info] celeborn-client-spark-3-shaded
+[info] celeborn-common
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-spark-common
+[info] celeborn-spark-group
+[info] celeborn-spark-it
+[info] celeborn-worker
+```
+
+To enable the Flink 1.15 client module, add `-Pflink-1.15`:
+
+```
+# ./build/sbt -Pflink-1.15 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-flink-1_15
+[info] celeborn-client-flink-1_15-shaded
+[info] celeborn-common
+[info] celeborn-flink-common
+[info] celeborn-flink-group
+[info] celeborn-flink-it
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-worker
+```
+
+By using these profiles, you can easily switch between different client modules for Spark and Flink. These profiles enable specific dependencies and configurations relevant to the chosen version. This way, you can conveniently manage and build the desired configurations of the Celeborn project.
+
+## Building Spark/Flink Assembly Client Jars
+
+For example, you can build the Spark 3.3 client assembly jar by running the following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-spark-3-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pspark-3.3` profile:
+$ ./build/sbt -Pspark-3.3 celeborn-client-spark-3-shaded/assembly
+```
+
+Similarly, you can build the Flink 1.15 client assembly jar using the following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-flink-1_15-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pflink-1.15` profile:
+$ ./build/sbt -Pflink-1.15 celeborn-client-flink-1_15-shaded/assembly
+```
+
+By executing these commands, you will create assembly jar files for the respective Spark and Flink client modules. The assembly jar bundles all the dependencies, allowing the client module to be used independently with all required dependencies included.
+
+## Building submodules individually
+
+For instance, you can build the Celeborn Master module using:
+
+```
+$ # sbt
+$ ./build/sbt
+> project celeborn-master
+> package
+
+$ # Or, you can build the celeborn-master module with sbt directly using:
+$ ./build/sbt celeborn-master/package
+```
+
+## Testing with SBT
+
+To run all tests for the Celeborn project, you can use the following command:
+
+```shell
+./build/sbt test
+```
+
+Running tests for specific versions of Spark/Flink client.
+
+For example, to run the test cases for the Spark 3.3 client, use the following command:
+
+```shell
+$ ./build/sbt -Pspark-3.3 test
+
+$ # only run spark client related modules tests
+$ ./build/sbt -Pspark-3.3 celeborn-spark-group/test
+```
+
+Similarly, to run the test cases for the Flink 1.15 client, use the following command:
+
+```shell
+$ ./build/sbt -Pflink-1.15 test
+
+$ # only run flink client related modules tests
+$ ./build/sbt -Pflink-1.15 celeborn-flink-group/test
+```
+
+## Running Individual Tests
+
+When developing locally, it’s often convenient to run a single test or a few tests, rather than running the entire test suite.
+
+The fastest way to run individual tests is to use the sbt console. It’s fastest to keep a sbt console open, and use it to re-run tests as necessary. For example, to run all of the tests in a particular project, e.g., master:
+
+```
+$ ./build/sbt
+> project celeborn-master
+> test
+```
+You can run a single test suite using the `testOnly` command. For example, to run the `SlotsAllocatorSuiteJ`:
+
+```
+> testOnly org.apache.celeborn.service.deploy.master.SlotsAllocatorSuiteJ
+```
+The `testOnly` command accepts wildcards; e.g., you can also run the `SlotsAllocatorSuiteJ` with:
+
+```
+> testOnly *SlotsAllocatorSuiteJ
+```
+Or you could run all of the tests in the `master` package:
+
+```
+> testOnly org.apache.celeborn.service.deploy.master.*
+```
+If you’d like to run just a single Java test in the `SlotsAllocatorSuiteJ`, e.g., a test that with the name `testAllocateSlotsForSinglePartitionId`, you run the following command in the sbt console:
+
+```
+> testOnly *SlotsAllocatorSuiteJ -- *SlotsAllocatorSuiteJ.testAllocateSlotsForSinglePartitionId
+```
+
+If you’d like to run just a single Scala test in the `AppDiskUsageMetricSuite`, e.g., a test that incudes "app usage snapshot" in the name, you run the following command in the sbt console:
+
+```
+> testOnly *AppDiskUsageMetricSuite -- -z "app usage snapshot"
+```
+
+If you’d prefer, you can run all of these commands on the command line (but this will be slower than running tests using an open console). To do this, you need to surround `testOnly` and the following arguments in quotes:
+
+```
+$ ./build/sbt "celeborn-master/testOnly *AppDiskUsageMetricSuite -- -z \"app usage snapshot\""
+```
+For more about how to run individual tests with sbt, see the [sbt documentation](https://www.scala-sbt.org/1.x/docs/Testing.html) and [JUnit Interface](https://github.com/sbt/junit-interface/#junit-interface).
+
+# Accelerating SBT
+
+This section provides instructions on setting up repository mirrors or proxies for a smoother SBT experience. Depending on your location and network conditions, you can choose the appropriate approach to accelerate SBT startup and enhance dependency retrieval.
+
+## Accelerating SBT Startup
+
+The SBT startup process involves fetching the SBT bootstrap jar, which is typically obtained from the Maven Central Repository (https://repo1.maven.org/maven2/). If you encounter slow access to this repository or if it's inaccessible in your network environment, you can expedite the SBT startup by configuring a custom artifact repository using the `DEFAULT_ARTIFACT_REPOSITORY` environment variable.
+
+
+```shell
+$ # The following command fetches sbt-launch-x.y.z.jar from https://maven.aliyun.com/nexus/content/groups/public/
+$ # Ensure that the URL ends with a trailing slash "/"
+$ export DEFAULT_ARTIFACT_REPOSITORY=https://maven.aliyun.com/nexus/content/groups/public/
+$ ./build/sbt
+```
+
+This will initiate SBT using the specified repository, allowing for faster download and startup times.
+
+## Custom SBT Repositories
+
+The current repositories embedded within the Celeborn project are detailed below:
+
+```
+[repositories]
+ local
+ mavenLocal: file://${user.home}/.m2/repository/
+ local-preloaded-ivy: file:///${sbt.preloaded-${sbt.global.base-${user.home}/.sbt}/preloaded/}, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext]
+ local-preloaded: file:///${sbt.preloaded-${sbt.global.base-${user.home}/.sbt}/preloaded/}
+ # The system property value of `celeborn.sbt.default.artifact.repository` is
+ # fetched from the environment variable `DEFAULT_ARTIFACT_REPOSITORY` and
+ # assigned within the build/sbt-launch-lib.bash script.
+ private: ${celeborn.sbt.default.artifact.repository-file:///dev/null}
+ gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/repos/central/data/
+ maven-central
+ typesafe-ivy-releases: https://repo.typesafe.com/typesafe/ivy-releases/, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext], bootOnly
+ sbt-ivy-snapshots: https://repo.scala-sbt.org/scalasbt/ivy-snapshots/, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext], bootOnly
+ sbt-plugin-releases: https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
+ bintray-typesafe-sbt-plugin-releases: https://dl.bintray.com/typesafe/sbt-plugins/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
+ bintray-spark-packages: https://dl.bintray.com/spark-packages/maven/
+ typesafe-releases: https://repo.typesafe.com/typesafe/releases/
+```
+
+For numerous developers across various regions, the default repository download speeds are less than optimal. To address this concern, we have curated a selection of verified public mirror templates tailored for specific regions with a significant local developer presence. For instance, we provide the `repositories-cn.template` template for developers situated within the expanse of the Chinese mainland, and the `repositories-asia.template` template designed for developers across the Asian continent. In such cases, the following command can be employed to enhance dependency download speeds:
+
+```
+cp build/sbt-config/repositories-asia.template build/sbt-config/repositories-local
+# Or
+cp build/sbt-config/repositories-cn.template build/sbt-config/repositories-local
Review Comment:
```suggestion
cp build/sbt-config/repositories-cn.template build/sbt-config/repositories-local
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1795: [CELEBORN-877][DOC] Document on SBT
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1795:
URL: https://github.com/apache/incubator-celeborn/pull/1795#discussion_r1290103203
##########
docs/developers/sbt.md:
##########
@@ -0,0 +1,276 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Building via SBT
+
+Starting from version 0.4.0, the Celeborn project supports building and packaging using SBT. This article provides a detailed guide on how to build the Celeborn project using SBT.
+
+# System Requirements
+
+Celeborn Service (master/worker) supports Scala 2.11/Scala 2.12 and Java 8/11/17.
Review Comment:
```suggestion
Celeborn Service (master/worker) supports Scala 2.11/2.12 and Java 8/11/17.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on pull request #1795: [CELEBORN-877][DOC] Document on SBT
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #1795:
URL: https://github.com/apache/incubator-celeborn/pull/1795#issuecomment-1673224391
cc @RexXiong @zhongqiangczq @FMX, I suppose it's friendly enough for new developers to learn the SBT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1795: [CELEBORN-877][DOC] Document on SBT
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1795:
URL: https://github.com/apache/incubator-celeborn/pull/1795#discussion_r1290194135
##########
docs/developers/sbt.md:
##########
@@ -0,0 +1,276 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Building via SBT
+
+Starting from version 0.4.0, the Celeborn project supports building and packaging using SBT. This article provides a detailed guide on how to build the Celeborn project using SBT.
+
+# System Requirements
+
+Celeborn Service (master/worker) supports Scala 2.11/Scala 2.12 and Java 8/11/17.
+
+The table following indicates the compatibility of Celeborn Spark and Flink clients with different versions of Spark and Flink for various Java and Scala versions:
+
+| | Java 8/Scala 2.11 | Java 8/Scala 2.12 | Java 11/Scala 2.12 | Java 17/Scala 2.12 |
+|----------------|-------------------|-------------------|--------------------|--------------------|
+| Spark 2.4 | ✔ | ❌ | ❌ | ❌ |
+| Spark 3.0 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.1 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.2 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.3 | ❌ | ✔ | ✔ | ✔ |
+| Spark 3.4 | ❌ | ✔ | ✔ | ✔ |
+| Flink 1.14 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.15 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.17 | ❌ | ✔ | ✔ | ❌ |
+
+# Useful SBT commands
+
+## Packaging the Project
+
+As an example, one can build a version of Celeborn as follows:
+
+```
+./build/sbt clean package
+```
+
+To create a Celeborn distribution like those distributed by the [Celeborn Downloads](https://celeborn.apache.org/download/) page, and that is laid out so as to be runnable, use `./build/make-distribution.sh` in the project root directory.
+
+```
+./build/make-distribution.sh --sbt
Review Comment:
maybe `--sbt-enabled`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1795: [CELEBORN-877][DOC] Document on SBT
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1795:
URL: https://github.com/apache/incubator-celeborn/pull/1795#discussion_r1287688113
##########
docs/sbt.md:
##########
@@ -0,0 +1,275 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
Review Comment:
indent and space lines
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1795: [CELEBORN-877][DOC] Document on SBT
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1795:
URL: https://github.com/apache/incubator-celeborn/pull/1795#discussion_r1290354455
##########
docs/developers/sbt.md:
##########
@@ -0,0 +1,277 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Building via SBT
+
+Starting from version 0.4.0, the Celeborn project supports building and packaging using SBT. This article provides a detailed guide on how to build the Celeborn project using SBT.
+
+# System Requirements
+
+Celeborn Service (master/worker) supports Scala 2.11/2.12 and Java 8/11/17.
+
+The following table indicates the compatibility of Celeborn Spark and Flink clients with different versions of Spark and Flink for various Java and Scala versions:
+
+| | Java 8/Scala 2.11 | Java 8/Scala 2.12 | Java 11/Scala 2.12 | Java 17/Scala 2.12 |
+|----------------|-------------------|-------------------|--------------------|--------------------|
+| Spark 2.4 | ✔ | ❌ | ❌ | ❌ |
+| Spark 3.0 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.1 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.2 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.3 | ❌ | ✔ | ✔ | ✔ |
+| Spark 3.4 | ❌ | ✔ | ✔ | ✔ |
+| Flink 1.14 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.15 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.17 | ❌ | ✔ | ✔ | ❌ |
+
+# Useful SBT commands
+
+## Packaging the Project
+
+As an example, one can build a version of Celeborn as follows:
+
+```
+./build/sbt clean package
+```
+
+To create a Celeborn distribution like those distributed by the [Celeborn Downloads](https://celeborn.apache.org/download/) page, and that is laid out so as to be runnable, use `./build/make-distribution.sh` in the project root directory.
+
+```
+./build/make-distribution.sh --sbt
+```
+
+## Maven-Style Profile Management
+
+We have adopted the Maven-style profile management for our Client module. For example, you can enable the Spark 3.3 client module by adding `-Pspark-3.3`:
+
+```
+# ./build/sbt -Pspark-3.3 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-spark-3
+[info] celeborn-client-spark-3-shaded
+[info] celeborn-common
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-spark-common
+[info] celeborn-spark-group
+[info] celeborn-spark-it
+[info] celeborn-worker
+```
+
+To enable the Flink 1.15 client module, add `-Pflink-1.15`:
+
+```
+# ./build/sbt -Pflink-1.15 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-flink-1_15
+[info] celeborn-client-flink-1_15-shaded
+[info] celeborn-common
+[info] celeborn-flink-common
+[info] celeborn-flink-group
+[info] celeborn-flink-it
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-worker
+```
+
+By using these profiles, you can easily switch between different client modules for Spark and Flink. These profiles enable specific dependencies and configurations relevant to the chosen version. This way, you can conveniently manage and build the desired configurations of the Celeborn project.
+
+## Building Spark/Flink Assembly Client Jars
+
+For example, you can build the Spark 3.3 client assembly jar by running the following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-spark-3-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pspark-3.3` profile:
+$ ./build/sbt -Pspark-3.3 celeborn-client-spark-3-shaded/assembly
+```
+
+Similarly, you can build the Flink 1.15 client assembly jar using the following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-flink-1_15-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pflink-1.15` profile:
+$ ./build/sbt -Pflink-1.15 celeborn-client-flink-1_15-shaded/assembly
+```
+
+By executing these commands, you will create assembly jar files for the respective Spark and Flink client modules. The assembly jar bundles all the dependencies, allowing the client module to be used independently with all required dependencies included.
+
+## Building submodules individually
+
+For instance, you can build the Celeborn Client module using:
+
+```
+$ # sbt
+$ build/sbt
+> project celeborn-client
+> package
+
+$ # Or, you can build the celeborn-client module with sbt directly using:
+$ build/sbt celeborn-client/package
+```
+
+## Testing with SBT
+
+To run all tests for the Celeborn project, you can use the following command:
+
+```shell
+./build/sbt test
+```
+
+Running tests for specific versions of Spark/Flink client.
+
+For example, to run the test cases for the Spark 3.3 client, use the following command:
+
+```shell
+$ ./build/sbt -Pspark-3.3 test
+
+$ # only run spark client related modules tests
+$ ./build/sbt -Pspark-3.3 celeborn-spark-group/test
+```
+
+Similarly, to run the test cases for the Flink 1.15 client, use the following command:
+
+```shell
+$ ./build/sbt -Pflink-1.15 test
+
+$ # only run flink client related modules tests
+$ ./build/sbt -Pflink-1.15 celeborn-flink-group/test
+```
+
+## Running Individual Tests
+
+When developing locally, it’s often convenient to run a single test or a few tests, rather than running the entire test suite.
+
+The fastest way to run individual tests is to use the sbt console. It’s fastest to keep a sbt console open, and use it to re-run tests as necessary. For example, to run all of the tests in a particular project, e.g., client:
+
+```
+$ build/sbt
+> project celeborn-client
+> test
+```
+You can run a single test suite using the `testOnly` command. For example, to run the `ShuffleClientSuiteJ`:
+
+```
+> testOnly org.apache.celeborn.client.ShuffleClientSuiteJ
+```
+The `testOnly` command accepts wildcards; e.g., you can also run the `ShuffleClientSuiteJ` with:
+
+```
+> testOnly *ShuffleClientSuiteJ
+```
+Or you could run all of the tests in the `client` package:
+
+```
+> testOnly org.apache.celeborn.client.*
+```
+If you’d like to run just a single Java test in the `ShuffleClientSuiteJ`, e.g., a test that with the name `testPushData`, you run the following command in the sbt console:
+
+```
+> testOnly *ShuffleClientSuiteJ -- *ShuffleClientSuiteJ.testPushData
+```
+
+If you’d like to run just a single Scala test in the `WorkerStatusTrackerSuite`, e.g., a test that incudes "handleHeartbeatResponse" in the name, you run the following command in the sbt console:
+
+```
+> testOnly *WorkerStatusTrackerSuite -- -z handleHeartbeatResponse
Review Comment:
can you use an example which contains space in the test name? IIRC, it's single/double quote sensitive
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1795: [CELEBORN-877][DOC] Document on SBT
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1795:
URL: https://github.com/apache/incubator-celeborn/pull/1795#discussion_r1290101840
##########
docs/cluster_planning.md:
##########
@@ -1,12 +1,14 @@
---
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
+ contributor license agreements. See the NOTICE file distributed with
Review Comment:
super nit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1795: [CELEBORN-877][DOC] Document on SBT
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1795:
URL: https://github.com/apache/incubator-celeborn/pull/1795#discussion_r1290105589
##########
docs/developers/sbt.md:
##########
@@ -0,0 +1,276 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Building via SBT
+
+Starting from version 0.4.0, the Celeborn project supports building and packaging using SBT. This article provides a detailed guide on how to build the Celeborn project using SBT.
+
+# System Requirements
+
+Celeborn Service (master/worker) supports Scala 2.11/Scala 2.12 and Java 8/11/17.
+
+The table following indicates the compatibility of Celeborn Spark and Flink clients with different versions of Spark and Flink for various Java and Scala versions:
+
+| | Java 8/Scala 2.11 | Java 8/Scala 2.12 | Java 11/Scala 2.12 | Java 17/Scala 2.12 |
+|----------------|-------------------|-------------------|--------------------|--------------------|
+| Spark 2.4 | ✔ | ❌ | ❌ | ❌ |
+| Spark 3.0 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.1 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.2 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.3 | ❌ | ✔ | ✔ | ✔ |
+| Spark 3.4 | ❌ | ✔ | ✔ | ✔ |
+| Flink 1.14 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.15 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.17 | ❌ | ✔ | ✔ | ❌ |
+
+# Useful SBT commands
+
+## Packaging the Project
+
+As an example, one can build a version of Celeborn as follows:
+
+```
+./build/sbt clean package
+```
+
+To create a Celeborn distribution like those distributed by the [Celeborn Downloads](https://celeborn.apache.org/download/) page, and that is laid out so as to be runnable, use `./build/make-distribution.sh` in the project root directory.
+
+```
+./build/make-distribution.sh --sbt
+```
+
+## Maven-Style Profile Management
+
+We have adopted the Maven-style profile management for our Client module. For example, you can enable the Spark 3.3 client module by adding `-Pspark-3.3`:
+
+```
+# ./build/sbt -Pspark-3.3 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-spark-3
+[info] celeborn-client-spark-3-shaded
+[info] celeborn-common
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-spark-common
+[info] celeborn-spark-group
+[info] celeborn-spark-it
+[info] celeborn-worker
+```
+
+To enable the Flink 1.15 client module, add `-Pflink-1.15`:
+
+```
+# ./build/sbt -Pflink-1.15 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-flink-1_15
+[info] celeborn-client-flink-1_15-shaded
+[info] celeborn-common
+[info] celeborn-flink-common
+[info] celeborn-flink-group
+[info] celeborn-flink-it
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-worker
+```
+
+By using these profiles, you can easily switch between different client modules for Spark and Flink. These profiles enable specific dependencies and configurations relevant to the chosen version. This way, you can conveniently manage and build the desired configurations of the Celeborn project.
+
+## Building Spark/Flink Assembly Client Jars
+
+For example, you can build the Spark 3.3 client assembly jar by running the following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-spark-3-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pspark-3.3` profile:
Review Comment:
break it down
##########
docs/developers/sbt.md:
##########
@@ -0,0 +1,276 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Building via SBT
+
+Starting from version 0.4.0, the Celeborn project supports building and packaging using SBT. This article provides a detailed guide on how to build the Celeborn project using SBT.
+
+# System Requirements
+
+Celeborn Service (master/worker) supports Scala 2.11/Scala 2.12 and Java 8/11/17.
+
+The table following indicates the compatibility of Celeborn Spark and Flink clients with different versions of Spark and Flink for various Java and Scala versions:
+
+| | Java 8/Scala 2.11 | Java 8/Scala 2.12 | Java 11/Scala 2.12 | Java 17/Scala 2.12 |
+|----------------|-------------------|-------------------|--------------------|--------------------|
+| Spark 2.4 | ✔ | ❌ | ❌ | ❌ |
+| Spark 3.0 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.1 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.2 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.3 | ❌ | ✔ | ✔ | ✔ |
+| Spark 3.4 | ❌ | ✔ | ✔ | ✔ |
+| Flink 1.14 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.15 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.17 | ❌ | ✔ | ✔ | ❌ |
+
+# Useful SBT commands
+
+## Packaging the Project
+
+As an example, one can build a version of Celeborn as follows:
+
+```
+./build/sbt clean package
+```
+
+To create a Celeborn distribution like those distributed by the [Celeborn Downloads](https://celeborn.apache.org/download/) page, and that is laid out so as to be runnable, use `./build/make-distribution.sh` in the project root directory.
+
+```
+./build/make-distribution.sh --sbt
+```
+
+## Maven-Style Profile Management
+
+We have adopted the Maven-style profile management for our Client module. For example, you can enable the Spark 3.3 client module by adding `-Pspark-3.3`:
+
+```
+# ./build/sbt -Pspark-3.3 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-spark-3
+[info] celeborn-client-spark-3-shaded
+[info] celeborn-common
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-spark-common
+[info] celeborn-spark-group
+[info] celeborn-spark-it
+[info] celeborn-worker
+```
+
+To enable the Flink 1.15 client module, add `-Pflink-1.15`:
+
+```
+# ./build/sbt -Pflink-1.15 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-flink-1_15
+[info] celeborn-client-flink-1_15-shaded
+[info] celeborn-common
+[info] celeborn-flink-common
+[info] celeborn-flink-group
+[info] celeborn-flink-it
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-worker
+```
+
+By using these profiles, you can easily switch between different client modules for Spark and Flink. These profiles enable specific dependencies and configurations relevant to the chosen version. This way, you can conveniently manage and build the desired configurations of the Celeborn project.
+
+## Building Spark/Flink Assembly Client Jars
+
+For example, you can build the Spark 3.3 client assembly jar by running the following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-spark-3-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pspark-3.3` profile:
+$ ./build/sbt -Pspark-3.3 celeborn-client-spark-3-shaded/assembly
+```
+
+Similarly, you can build the Flink 1.15 client assembly jar using the following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-flink-1_15-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pflink-1.15` profile:
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on pull request #1795: [CELEBORN-877][DOC] Document on SBT
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #1795:
URL: https://github.com/apache/incubator-celeborn/pull/1795#issuecomment-1673222782
Good job! The article is clear and detailed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1795: [CELEBORN-877][DOC] Document on SBT
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1795:
URL: https://github.com/apache/incubator-celeborn/pull/1795#discussion_r1290171009
##########
docs/developers/sbt.md:
##########
@@ -0,0 +1,276 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Building via SBT
+
+Starting from version 0.4.0, the Celeborn project supports building and packaging using SBT. This article provides a detailed guide on how to build the Celeborn project using SBT.
+
+# System Requirements
+
+Celeborn Service (master/worker) supports Scala 2.11/Scala 2.12 and Java 8/11/17.
+
+The table following indicates the compatibility of Celeborn Spark and Flink clients with different versions of Spark and Flink for various Java and Scala versions:
+
+| | Java 8/Scala 2.11 | Java 8/Scala 2.12 | Java 11/Scala 2.12 | Java 17/Scala 2.12 |
+|----------------|-------------------|-------------------|--------------------|--------------------|
+| Spark 2.4 | ✔ | ❌ | ❌ | ❌ |
+| Spark 3.0 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.1 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.2 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.3 | ❌ | ✔ | ✔ | ✔ |
+| Spark 3.4 | ❌ | ✔ | ✔ | ✔ |
+| Flink 1.14 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.15 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.17 | ❌ | ✔ | ✔ | ❌ |
+
+# Useful SBT commands
+
+## Packaging the Project
+
+As an example, one can build a version of Celeborn as follows:
+
+```
+./build/sbt clean package
+```
+
+To create a Celeborn distribution like those distributed by the [Celeborn Downloads](https://celeborn.apache.org/download/) page, and that is laid out so as to be runnable, use `./build/make-distribution.sh` in the project root directory.
+
+```
+./build/make-distribution.sh --sbt
+```
+
+## Maven-Style Profile Management
+
+We have adopted the Maven-style profile management for our Client module. For example, you can enable the Spark 3.3 client module by adding `-Pspark-3.3`:
+
+```
+# ./build/sbt -Pspark-3.3 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-spark-3
+[info] celeborn-client-spark-3-shaded
+[info] celeborn-common
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-spark-common
+[info] celeborn-spark-group
+[info] celeborn-spark-it
+[info] celeborn-worker
+```
+
+To enable the Flink 1.15 client module, add `-Pflink-1.15`:
+
+```
+# ./build/sbt -Pflink-1.15 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-flink-1_15
+[info] celeborn-client-flink-1_15-shaded
+[info] celeborn-common
+[info] celeborn-flink-common
+[info] celeborn-flink-group
+[info] celeborn-flink-it
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-worker
+```
+
+By using these profiles, you can easily switch between different client modules for Spark and Flink. These profiles enable specific dependencies and configurations relevant to the chosen version. This way, you can conveniently manage and build the desired configurations of the Celeborn project.
+
+## Building Spark/Flink Assembly Client Jars
+
+For example, you can build the Spark 3.3 client assembly jar by running the following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-spark-3-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pspark-3.3` profile:
+$ ./build/sbt -Pspark-3.3 celeborn-client-spark-3-shaded/assembly
+```
+
+Similarly, you can build the Flink 1.15 client assembly jar using the following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-flink-1_15-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pflink-1.15` profile:
+$ ./build/sbt -Pflink-1.15 celeborn-client-flink-1_15-shaded/assembly
+```
+
+By executing these commands, you will create assembly jar files for the respective Spark and Flink client modules. The assembly jar bundles all the dependencies, allowing the client module to be used independently with all required dependencies included.
+
+## Building submodules individually
+
+For instance, you can build the Celeborn Client module using:
+
+```
+$ # sbt
+$ build/sbt
+> project celeborn-client
+> package
+
+$ # Or, you can build the celeborn-client module with sbt directly using:
+$ build/sbt celeborn-client/package
+```
+
+## Testing with SBT
+
+To run all tests for the Celeborn project, you can use the following command:
+
+```shell
+./build/sbt test
+```
+
+Running tests for specific versions of Spark/Flink client.
+
+For example, to run the test cases for the Spark 3.3 client, use the following command:
+
+```shell
+$ ./build/sbt -Pspark-3.3 test
+
+$ # only run spark client related modules tests
+$ ./build/sbt -Pspark-3.3 celeborn-spark-group/test
+```
+
+Similarly, to run the test cases for the Flink 1.15 client, use the following command:
+
+```shell
+$ ./build/sbt -Pflink-1.15 test
+
+$ # only run flink client related modules tests
+$ ./build/sbt -Pflink-1.15 celeborn-flink-group/test
+```
+
+## Running Individual Tests
+
+When developing locally, it’s often convenient to run a single test or a few tests, rather than running the entire test suite.
+
+The fastest way to run individual tests is to use the sbt console. It’s fastest to keep a sbt console open, and use it to re-run tests as necessary. For example, to run all of the tests in a particular project, e.g., client:
+
+```
+$ build/sbt
+> project celeborn-client
+> test
+```
+You can run a single test suite using the `testOnly` command. For example, to run the `ShuffleClientSuiteJ`:
+
+```
+> testOnly org.apache.celeborn.client.ShuffleClientSuiteJ
+```
+The `testOnly` command accepts wildcards; e.g., you can also run the `ShuffleClientSuiteJ` with:
+
+```
+> testOnly *ShuffleClientSuiteJ
+```
+Or you could run all of the tests in the `client` package:
+
+```
+> testOnly org.apache.celeborn.client.*
+```
+If you’d like to run just a single Java test in the `ShuffleClientSuiteJ`, e.g., a test that with the name `testPushData`, you run the following command in the sbt console:
+
+```
+> testOnly *ShuffleClientSuiteJ -- *ShuffleClientSuiteJ.testPushData
+```
+
+If you’d like to run just a single Scala test in the `WorkerStatusTrackerSuite`, e.g., a test that incudes "handleHeartbeatResponse" in the name, you run the following command in the sbt console:
+
+```
+> testOnly *WorkerStatusTrackerSuite -- -z handleHeartbeatResponse
+```
+
+If you’d prefer, you can run all of these commands on the command line (but this will be slower than running tests using an open console). To do this, you need to surround `testOnly` and the following arguments in quotes:
+
+```
+$ build/sbt "celeborn-client/testOnly *WorkerStatusTrackerSuite -- -z handleHeartbeatResponse"
+```
+For more about how to run individual tests with sbt, see the [sbt documentation](https://www.scala-sbt.org/1.x/docs/Testing.html) and [JUnit Interface](https://github.com/sbt/junit-interface/#junit-interface).
+
+# Accelerating SBT
+
+This section provides instructions on setting up repository proxies for a smoother SBT experience. Depending on your location and network conditions, you can choose the appropriate approach to accelerate SBT startup and enhance dependency retrieval.
+
+## Accelerating SBT Startup
+
+The SBT startup process involves fetching the SBT bootstrap jar, which is typically obtained from the Maven Central Repository (https://repo1.maven.org/maven2/). If you encounter slow access to this repository or if it's inaccessible in your network environment, you can expedite the SBT startup by configuring a custom artifact repository using the `DEFAULT_ARTIFACT_REPOSITORY` environment variable.
+
+
+```shell
+$ # The following command fetches sbt-launch-x.y.z.jar from https://maven.aliyun.com/nexus/content/groups/public/
+$ # Ensure that the URL ends with a trailing slash "/"
+$ export DEFAULT_ARTIFACT_REPOSITORY=https://maven.aliyun.com/nexus/content/groups/public/ && ./build/sbt
+```
+
+This will initiate SBT using the specified repository, allowing for faster download and startup times.
+
+## Custom SBT Repositories
+
+The current repositories embedded within the Celeborn project are detailed below:
+
+```
+[repositories]
+ local
+ mavenLocal: file://${user.home}/.m2/repository/
+ local-preloaded-ivy: file:///${sbt.preloaded-${sbt.global.base-${user.home}/.sbt}/preloaded/}, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext]
+ local-preloaded: file:///${sbt.preloaded-${sbt.global.base-${user.home}/.sbt}/preloaded/}
+ # The system property value of `celeborn.sbt.default.artifact.repository` is
+ # fetched from the environment variable `DEFAULT_ARTIFACT_REPOSITORY` and
+ # assigned within the build/sbt-launch-lib.bash script.
+ private: ${celeborn.sbt.default.artifact.repository-file:///dev/null}
+ gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/repos/central/data/
+ maven-central
+ typesafe-ivy-releases: https://repo.typesafe.com/typesafe/ivy-releases/, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext], bootOnly
+ sbt-ivy-snapshots: https://repo.scala-sbt.org/scalasbt/ivy-snapshots/, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext], bootOnly
+ sbt-plugin-releases: https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
+ bintray-typesafe-sbt-plugin-releases: https://dl.bintray.com/typesafe/sbt-plugins/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
+ bintray-spark-packages: https://dl.bintray.com/spark-packages/maven/
+ typesafe-releases: https://repo.typesafe.com/typesafe/releases/
+```
+
+For numerous developers across various regions, the default repository download speeds are less than optimal. Consider, for example, developers situated in mainland China. In such cases, the following command can be employed to enhance dependency download speeds:
+
+```
+cp build/sbt-config/repositories-cn.template build/sbt-config/repositories-local
+```
+
+> NOTE: <br>
Review Comment:
yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1795: [CELEBORN-877][DOC] Document on SBT
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1795:
URL: https://github.com/apache/incubator-celeborn/pull/1795#discussion_r1290103692
##########
docs/developers/sbt.md:
##########
@@ -0,0 +1,276 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Building via SBT
+
+Starting from version 0.4.0, the Celeborn project supports building and packaging using SBT. This article provides a detailed guide on how to build the Celeborn project using SBT.
+
+# System Requirements
+
+Celeborn Service (master/worker) supports Scala 2.11/Scala 2.12 and Java 8/11/17.
+
+The table following indicates the compatibility of Celeborn Spark and Flink clients with different versions of Spark and Flink for various Java and Scala versions:
Review Comment:
```suggestion
The following table indicates the compatibility of Celeborn Spark and Flink clients with different versions of Spark and Flink for various Java and Scala versions:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1795: [CELEBORN-877][DOC] Document on SBT
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1795:
URL: https://github.com/apache/incubator-celeborn/pull/1795#discussion_r1290171009
##########
docs/developers/sbt.md:
##########
@@ -0,0 +1,276 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Building via SBT
+
+Starting from version 0.4.0, the Celeborn project supports building and packaging using SBT. This article provides a detailed guide on how to build the Celeborn project using SBT.
+
+# System Requirements
+
+Celeborn Service (master/worker) supports Scala 2.11/Scala 2.12 and Java 8/11/17.
+
+The table following indicates the compatibility of Celeborn Spark and Flink clients with different versions of Spark and Flink for various Java and Scala versions:
+
+| | Java 8/Scala 2.11 | Java 8/Scala 2.12 | Java 11/Scala 2.12 | Java 17/Scala 2.12 |
+|----------------|-------------------|-------------------|--------------------|--------------------|
+| Spark 2.4 | ✔ | ❌ | ❌ | ❌ |
+| Spark 3.0 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.1 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.2 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.3 | ❌ | ✔ | ✔ | ✔ |
+| Spark 3.4 | ❌ | ✔ | ✔ | ✔ |
+| Flink 1.14 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.15 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.17 | ❌ | ✔ | ✔ | ❌ |
+
+# Useful SBT commands
+
+## Packaging the Project
+
+As an example, one can build a version of Celeborn as follows:
+
+```
+./build/sbt clean package
+```
+
+To create a Celeborn distribution like those distributed by the [Celeborn Downloads](https://celeborn.apache.org/download/) page, and that is laid out so as to be runnable, use `./build/make-distribution.sh` in the project root directory.
+
+```
+./build/make-distribution.sh --sbt
+```
+
+## Maven-Style Profile Management
+
+We have adopted the Maven-style profile management for our Client module. For example, you can enable the Spark 3.3 client module by adding `-Pspark-3.3`:
+
+```
+# ./build/sbt -Pspark-3.3 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-spark-3
+[info] celeborn-client-spark-3-shaded
+[info] celeborn-common
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-spark-common
+[info] celeborn-spark-group
+[info] celeborn-spark-it
+[info] celeborn-worker
+```
+
+To enable the Flink 1.15 client module, add `-Pflink-1.15`:
+
+```
+# ./build/sbt -Pflink-1.15 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-flink-1_15
+[info] celeborn-client-flink-1_15-shaded
+[info] celeborn-common
+[info] celeborn-flink-common
+[info] celeborn-flink-group
+[info] celeborn-flink-it
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-worker
+```
+
+By using these profiles, you can easily switch between different client modules for Spark and Flink. These profiles enable specific dependencies and configurations relevant to the chosen version. This way, you can conveniently manage and build the desired configurations of the Celeborn project.
+
+## Building Spark/Flink Assembly Client Jars
+
+For example, you can build the Spark 3.3 client assembly jar by running the following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-spark-3-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pspark-3.3` profile:
+$ ./build/sbt -Pspark-3.3 celeborn-client-spark-3-shaded/assembly
+```
+
+Similarly, you can build the Flink 1.15 client assembly jar using the following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-flink-1_15-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pflink-1.15` profile:
+$ ./build/sbt -Pflink-1.15 celeborn-client-flink-1_15-shaded/assembly
+```
+
+By executing these commands, you will create assembly jar files for the respective Spark and Flink client modules. The assembly jar bundles all the dependencies, allowing the client module to be used independently with all required dependencies included.
+
+## Building submodules individually
+
+For instance, you can build the Celeborn Client module using:
+
+```
+$ # sbt
+$ build/sbt
+> project celeborn-client
+> package
+
+$ # Or, you can build the celeborn-client module with sbt directly using:
+$ build/sbt celeborn-client/package
+```
+
+## Testing with SBT
+
+To run all tests for the Celeborn project, you can use the following command:
+
+```shell
+./build/sbt test
+```
+
+Running tests for specific versions of Spark/Flink client.
+
+For example, to run the test cases for the Spark 3.3 client, use the following command:
+
+```shell
+$ ./build/sbt -Pspark-3.3 test
+
+$ # only run spark client related modules tests
+$ ./build/sbt -Pspark-3.3 celeborn-spark-group/test
+```
+
+Similarly, to run the test cases for the Flink 1.15 client, use the following command:
+
+```shell
+$ ./build/sbt -Pflink-1.15 test
+
+$ # only run flink client related modules tests
+$ ./build/sbt -Pflink-1.15 celeborn-flink-group/test
+```
+
+## Running Individual Tests
+
+When developing locally, it’s often convenient to run a single test or a few tests, rather than running the entire test suite.
+
+The fastest way to run individual tests is to use the sbt console. It’s fastest to keep a sbt console open, and use it to re-run tests as necessary. For example, to run all of the tests in a particular project, e.g., client:
+
+```
+$ build/sbt
+> project celeborn-client
+> test
+```
+You can run a single test suite using the `testOnly` command. For example, to run the `ShuffleClientSuiteJ`:
+
+```
+> testOnly org.apache.celeborn.client.ShuffleClientSuiteJ
+```
+The `testOnly` command accepts wildcards; e.g., you can also run the `ShuffleClientSuiteJ` with:
+
+```
+> testOnly *ShuffleClientSuiteJ
+```
+Or you could run all of the tests in the `client` package:
+
+```
+> testOnly org.apache.celeborn.client.*
+```
+If you’d like to run just a single Java test in the `ShuffleClientSuiteJ`, e.g., a test that with the name `testPushData`, you run the following command in the sbt console:
+
+```
+> testOnly *ShuffleClientSuiteJ -- *ShuffleClientSuiteJ.testPushData
+```
+
+If you’d like to run just a single Scala test in the `WorkerStatusTrackerSuite`, e.g., a test that incudes "handleHeartbeatResponse" in the name, you run the following command in the sbt console:
+
+```
+> testOnly *WorkerStatusTrackerSuite -- -z handleHeartbeatResponse
+```
+
+If you’d prefer, you can run all of these commands on the command line (but this will be slower than running tests using an open console). To do this, you need to surround `testOnly` and the following arguments in quotes:
+
+```
+$ build/sbt "celeborn-client/testOnly *WorkerStatusTrackerSuite -- -z handleHeartbeatResponse"
+```
+For more about how to run individual tests with sbt, see the [sbt documentation](https://www.scala-sbt.org/1.x/docs/Testing.html) and [JUnit Interface](https://github.com/sbt/junit-interface/#junit-interface).
+
+# Accelerating SBT
+
+This section provides instructions on setting up repository proxies for a smoother SBT experience. Depending on your location and network conditions, you can choose the appropriate approach to accelerate SBT startup and enhance dependency retrieval.
+
+## Accelerating SBT Startup
+
+The SBT startup process involves fetching the SBT bootstrap jar, which is typically obtained from the Maven Central Repository (https://repo1.maven.org/maven2/). If you encounter slow access to this repository or if it's inaccessible in your network environment, you can expedite the SBT startup by configuring a custom artifact repository using the `DEFAULT_ARTIFACT_REPOSITORY` environment variable.
+
+
+```shell
+$ # The following command fetches sbt-launch-x.y.z.jar from https://maven.aliyun.com/nexus/content/groups/public/
+$ # Ensure that the URL ends with a trailing slash "/"
+$ export DEFAULT_ARTIFACT_REPOSITORY=https://maven.aliyun.com/nexus/content/groups/public/ && ./build/sbt
+```
+
+This will initiate SBT using the specified repository, allowing for faster download and startup times.
+
+## Custom SBT Repositories
+
+The current repositories embedded within the Celeborn project are detailed below:
+
+```
+[repositories]
+ local
+ mavenLocal: file://${user.home}/.m2/repository/
+ local-preloaded-ivy: file:///${sbt.preloaded-${sbt.global.base-${user.home}/.sbt}/preloaded/}, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext]
+ local-preloaded: file:///${sbt.preloaded-${sbt.global.base-${user.home}/.sbt}/preloaded/}
+ # The system property value of `celeborn.sbt.default.artifact.repository` is
+ # fetched from the environment variable `DEFAULT_ARTIFACT_REPOSITORY` and
+ # assigned within the build/sbt-launch-lib.bash script.
+ private: ${celeborn.sbt.default.artifact.repository-file:///dev/null}
+ gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/repos/central/data/
+ maven-central
+ typesafe-ivy-releases: https://repo.typesafe.com/typesafe/ivy-releases/, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext], bootOnly
+ sbt-ivy-snapshots: https://repo.scala-sbt.org/scalasbt/ivy-snapshots/, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext], bootOnly
+ sbt-plugin-releases: https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
+ bintray-typesafe-sbt-plugin-releases: https://dl.bintray.com/typesafe/sbt-plugins/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
+ bintray-spark-packages: https://dl.bintray.com/spark-packages/maven/
+ typesafe-releases: https://repo.typesafe.com/typesafe/releases/
+```
+
+For numerous developers across various regions, the default repository download speeds are less than optimal. Consider, for example, developers situated in mainland China. In such cases, the following command can be employed to enhance dependency download speeds:
+
+```
+cp build/sbt-config/repositories-cn.template build/sbt-config/repositories-local
+```
+
+> NOTE: <br>
Review Comment:
yep, if there is no BR, it will look like this:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1795: [CELEBORN-877][DOC] Document on SBT
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1795:
URL: https://github.com/apache/incubator-celeborn/pull/1795#discussion_r1290859189
##########
docs/developers/sbt.md:
##########
@@ -0,0 +1,277 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Building via SBT
+
+Starting from version 0.4.0, the Celeborn project supports building and packaging using SBT. This article provides a detailed guide on how to build the Celeborn project using SBT.
+
+# System Requirements
+
+Celeborn Service (master/worker) supports Scala 2.11/2.12 and Java 8/11/17.
+
+The following table indicates the compatibility of Celeborn Spark and Flink clients with different versions of Spark and Flink for various Java and Scala versions:
+
+| | Java 8/Scala 2.11 | Java 8/Scala 2.12 | Java 11/Scala 2.12 | Java 17/Scala 2.12 |
+|----------------|-------------------|-------------------|--------------------|--------------------|
+| Spark 2.4 | ✔ | ❌ | ❌ | ❌ |
+| Spark 3.0 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.1 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.2 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.3 | ❌ | ✔ | ✔ | ✔ |
+| Spark 3.4 | ❌ | ✔ | ✔ | ✔ |
+| Flink 1.14 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.15 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.17 | ❌ | ✔ | ✔ | ❌ |
+
+# Useful SBT commands
+
+## Packaging the Project
+
+As an example, one can build a version of Celeborn as follows:
+
+```
+./build/sbt clean package
+```
+
+To create a Celeborn distribution like those distributed by the [Celeborn Downloads](https://celeborn.apache.org/download/) page, and that is laid out so as to be runnable, use `./build/make-distribution.sh` in the project root directory.
+
+```
+./build/make-distribution.sh --sbt
+```
+
+## Maven-Style Profile Management
+
+We have adopted the Maven-style profile management for our Client module. For example, you can enable the Spark 3.3 client module by adding `-Pspark-3.3`:
+
+```
+# ./build/sbt -Pspark-3.3 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-spark-3
+[info] celeborn-client-spark-3-shaded
+[info] celeborn-common
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-spark-common
+[info] celeborn-spark-group
+[info] celeborn-spark-it
+[info] celeborn-worker
+```
+
+To enable the Flink 1.15 client module, add `-Pflink-1.15`:
+
+```
+# ./build/sbt -Pflink-1.15 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-flink-1_15
+[info] celeborn-client-flink-1_15-shaded
+[info] celeborn-common
+[info] celeborn-flink-common
+[info] celeborn-flink-group
+[info] celeborn-flink-it
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-worker
+```
+
+By using these profiles, you can easily switch between different client modules for Spark and Flink. These profiles enable specific dependencies and configurations relevant to the chosen version. This way, you can conveniently manage and build the desired configurations of the Celeborn project.
+
+## Building Spark/Flink Assembly Client Jars
+
+For example, you can build the Spark 3.3 client assembly jar by running the following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-spark-3-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pspark-3.3` profile:
+$ ./build/sbt -Pspark-3.3 celeborn-client-spark-3-shaded/assembly
+```
+
+Similarly, you can build the Flink 1.15 client assembly jar using the following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-flink-1_15-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pflink-1.15` profile:
+$ ./build/sbt -Pflink-1.15 celeborn-client-flink-1_15-shaded/assembly
+```
+
+By executing these commands, you will create assembly jar files for the respective Spark and Flink client modules. The assembly jar bundles all the dependencies, allowing the client module to be used independently with all required dependencies included.
+
+## Building submodules individually
+
+For instance, you can build the Celeborn Client module using:
+
+```
+$ # sbt
+$ build/sbt
+> project celeborn-client
+> package
+
+$ # Or, you can build the celeborn-client module with sbt directly using:
+$ build/sbt celeborn-client/package
+```
+
+## Testing with SBT
+
+To run all tests for the Celeborn project, you can use the following command:
+
+```shell
+./build/sbt test
+```
+
+Running tests for specific versions of Spark/Flink client.
+
+For example, to run the test cases for the Spark 3.3 client, use the following command:
+
+```shell
+$ ./build/sbt -Pspark-3.3 test
+
+$ # only run spark client related modules tests
+$ ./build/sbt -Pspark-3.3 celeborn-spark-group/test
+```
+
+Similarly, to run the test cases for the Flink 1.15 client, use the following command:
+
+```shell
+$ ./build/sbt -Pflink-1.15 test
+
+$ # only run flink client related modules tests
+$ ./build/sbt -Pflink-1.15 celeborn-flink-group/test
+```
+
+## Running Individual Tests
+
+When developing locally, it’s often convenient to run a single test or a few tests, rather than running the entire test suite.
+
+The fastest way to run individual tests is to use the sbt console. It’s fastest to keep a sbt console open, and use it to re-run tests as necessary. For example, to run all of the tests in a particular project, e.g., client:
+
+```
+$ build/sbt
+> project celeborn-client
+> test
+```
+You can run a single test suite using the `testOnly` command. For example, to run the `ShuffleClientSuiteJ`:
+
+```
+> testOnly org.apache.celeborn.client.ShuffleClientSuiteJ
+```
+The `testOnly` command accepts wildcards; e.g., you can also run the `ShuffleClientSuiteJ` with:
+
+```
+> testOnly *ShuffleClientSuiteJ
+```
+Or you could run all of the tests in the `client` package:
+
+```
+> testOnly org.apache.celeborn.client.*
+```
+If you’d like to run just a single Java test in the `ShuffleClientSuiteJ`, e.g., a test that with the name `testPushData`, you run the following command in the sbt console:
+
+```
+> testOnly *ShuffleClientSuiteJ -- *ShuffleClientSuiteJ.testPushData
+```
+
+If you’d like to run just a single Scala test in the `WorkerStatusTrackerSuite`, e.g., a test that incudes "handleHeartbeatResponse" in the name, you run the following command in the sbt console:
+
+```
+> testOnly *WorkerStatusTrackerSuite -- -z handleHeartbeatResponse
+```
+
+If you’d prefer, you can run all of these commands on the command line (but this will be slower than running tests using an open console). To do this, you need to surround `testOnly` and the following arguments in quotes:
+
+```
+$ build/sbt "celeborn-client/testOnly *WorkerStatusTrackerSuite -- -z handleHeartbeatResponse"
+```
+For more about how to run individual tests with sbt, see the [sbt documentation](https://www.scala-sbt.org/1.x/docs/Testing.html) and [JUnit Interface](https://github.com/sbt/junit-interface/#junit-interface).
+
+# Accelerating SBT
Review Comment:
after thought, I think it's fine, we can put other content in the future, like how to skip unnecessary checks, run tests in parallel，etc.
##########
docs/developers/sbt.md:
##########
@@ -0,0 +1,277 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Building via SBT
+
+Starting from version 0.4.0, the Celeborn project supports building and packaging using SBT. This article provides a detailed guide on how to build the Celeborn project using SBT.
+
+# System Requirements
+
+Celeborn Service (master/worker) supports Scala 2.11/2.12 and Java 8/11/17.
+
+The following table indicates the compatibility of Celeborn Spark and Flink clients with different versions of Spark and Flink for various Java and Scala versions:
+
+| | Java 8/Scala 2.11 | Java 8/Scala 2.12 | Java 11/Scala 2.12 | Java 17/Scala 2.12 |
+|----------------|-------------------|-------------------|--------------------|--------------------|
+| Spark 2.4 | ✔ | ❌ | ❌ | ❌ |
+| Spark 3.0 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.1 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.2 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.3 | ❌ | ✔ | ✔ | ✔ |
+| Spark 3.4 | ❌ | ✔ | ✔ | ✔ |
+| Flink 1.14 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.15 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.17 | ❌ | ✔ | ✔ | ❌ |
+
+# Useful SBT commands
+
+## Packaging the Project
+
+As an example, one can build a version of Celeborn as follows:
+
+```
+./build/sbt clean package
+```
+
+To create a Celeborn distribution like those distributed by the [Celeborn Downloads](https://celeborn.apache.org/download/) page, and that is laid out so as to be runnable, use `./build/make-distribution.sh` in the project root directory.
+
+```
+./build/make-distribution.sh --sbt
+```
+
+## Maven-Style Profile Management
+
+We have adopted the Maven-style profile management for our Client module. For example, you can enable the Spark 3.3 client module by adding `-Pspark-3.3`:
+
+```
+# ./build/sbt -Pspark-3.3 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-spark-3
+[info] celeborn-client-spark-3-shaded
+[info] celeborn-common
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-spark-common
+[info] celeborn-spark-group
+[info] celeborn-spark-it
+[info] celeborn-worker
+```
+
+To enable the Flink 1.15 client module, add `-Pflink-1.15`:
+
+```
+# ./build/sbt -Pflink-1.15 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-flink-1_15
+[info] celeborn-client-flink-1_15-shaded
+[info] celeborn-common
+[info] celeborn-flink-common
+[info] celeborn-flink-group
+[info] celeborn-flink-it
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-worker
+```
+
+By using these profiles, you can easily switch between different client modules for Spark and Flink. These profiles enable specific dependencies and configurations relevant to the chosen version. This way, you can conveniently manage and build the desired configurations of the Celeborn project.
+
+## Building Spark/Flink Assembly Client Jars
+
+For example, you can build the Spark 3.3 client assembly jar by running the following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-spark-3-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pspark-3.3` profile:
+$ ./build/sbt -Pspark-3.3 celeborn-client-spark-3-shaded/assembly
+```
+
+Similarly, you can build the Flink 1.15 client assembly jar using the following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-flink-1_15-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pflink-1.15` profile:
+$ ./build/sbt -Pflink-1.15 celeborn-client-flink-1_15-shaded/assembly
+```
+
+By executing these commands, you will create assembly jar files for the respective Spark and Flink client modules. The assembly jar bundles all the dependencies, allowing the client module to be used independently with all required dependencies included.
+
+## Building submodules individually
+
+For instance, you can build the Celeborn Client module using:
+
+```
+$ # sbt
+$ build/sbt
+> project celeborn-client
+> package
+
+$ # Or, you can build the celeborn-client module with sbt directly using:
+$ build/sbt celeborn-client/package
+```
+
+## Testing with SBT
+
+To run all tests for the Celeborn project, you can use the following command:
+
+```shell
+./build/sbt test
+```
+
+Running tests for specific versions of Spark/Flink client.
+
+For example, to run the test cases for the Spark 3.3 client, use the following command:
+
+```shell
+$ ./build/sbt -Pspark-3.3 test
+
+$ # only run spark client related modules tests
+$ ./build/sbt -Pspark-3.3 celeborn-spark-group/test
+```
+
+Similarly, to run the test cases for the Flink 1.15 client, use the following command:
+
+```shell
+$ ./build/sbt -Pflink-1.15 test
+
+$ # only run flink client related modules tests
+$ ./build/sbt -Pflink-1.15 celeborn-flink-group/test
+```
+
+## Running Individual Tests
+
+When developing locally, it’s often convenient to run a single test or a few tests, rather than running the entire test suite.
+
+The fastest way to run individual tests is to use the sbt console. It’s fastest to keep a sbt console open, and use it to re-run tests as necessary. For example, to run all of the tests in a particular project, e.g., client:
+
+```
+$ build/sbt
+> project celeborn-client
+> test
+```
+You can run a single test suite using the `testOnly` command. For example, to run the `ShuffleClientSuiteJ`:
+
+```
+> testOnly org.apache.celeborn.client.ShuffleClientSuiteJ
+```
+The `testOnly` command accepts wildcards; e.g., you can also run the `ShuffleClientSuiteJ` with:
+
+```
+> testOnly *ShuffleClientSuiteJ
+```
+Or you could run all of the tests in the `client` package:
+
+```
+> testOnly org.apache.celeborn.client.*
+```
+If you’d like to run just a single Java test in the `ShuffleClientSuiteJ`, e.g., a test that with the name `testPushData`, you run the following command in the sbt console:
+
+```
+> testOnly *ShuffleClientSuiteJ -- *ShuffleClientSuiteJ.testPushData
+```
+
+If you’d like to run just a single Scala test in the `WorkerStatusTrackerSuite`, e.g., a test that incudes "handleHeartbeatResponse" in the name, you run the following command in the sbt console:
+
+```
+> testOnly *WorkerStatusTrackerSuite -- -z handleHeartbeatResponse
+```
+
+If you’d prefer, you can run all of these commands on the command line (but this will be slower than running tests using an open console). To do this, you need to surround `testOnly` and the following arguments in quotes:
+
+```
+$ build/sbt "celeborn-client/testOnly *WorkerStatusTrackerSuite -- -z handleHeartbeatResponse"
+```
+For more about how to run individual tests with sbt, see the [sbt documentation](https://www.scala-sbt.org/1.x/docs/Testing.html) and [JUnit Interface](https://github.com/sbt/junit-interface/#junit-interface).
+
+# Accelerating SBT
Review Comment:
after thought, I think it's fine, we can put other content in the future, like how to skip unnecessary checks, run tests in parallel，etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1795: [CELEBORN-877][DOC] Document on SBT
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1795:
URL: https://github.com/apache/incubator-celeborn/pull/1795#discussion_r1290844078
##########
docs/developers/sbt.md:
##########
@@ -0,0 +1,277 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Building via SBT
+
+Starting from version 0.4.0, the Celeborn project supports building and packaging using SBT. This article provides a detailed guide on how to build the Celeborn project using SBT.
+
+# System Requirements
+
+Celeborn Service (master/worker) supports Scala 2.11/2.12 and Java 8/11/17.
+
+The following table indicates the compatibility of Celeborn Spark and Flink clients with different versions of Spark and Flink for various Java and Scala versions:
+
+| | Java 8/Scala 2.11 | Java 8/Scala 2.12 | Java 11/Scala 2.12 | Java 17/Scala 2.12 |
+|----------------|-------------------|-------------------|--------------------|--------------------|
+| Spark 2.4 | ✔ | ❌ | ❌ | ❌ |
+| Spark 3.0 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.1 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.2 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.3 | ❌ | ✔ | ✔ | ✔ |
+| Spark 3.4 | ❌ | ✔ | ✔ | ✔ |
+| Flink 1.14 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.15 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.17 | ❌ | ✔ | ✔ | ❌ |
+
+# Useful SBT commands
+
+## Packaging the Project
+
+As an example, one can build a version of Celeborn as follows:
+
+```
+./build/sbt clean package
+```
+
+To create a Celeborn distribution like those distributed by the [Celeborn Downloads](https://celeborn.apache.org/download/) page, and that is laid out so as to be runnable, use `./build/make-distribution.sh` in the project root directory.
+
+```
+./build/make-distribution.sh --sbt
+```
+
+## Maven-Style Profile Management
+
+We have adopted the Maven-style profile management for our Client module. For example, you can enable the Spark 3.3 client module by adding `-Pspark-3.3`:
+
+```
+# ./build/sbt -Pspark-3.3 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-spark-3
+[info] celeborn-client-spark-3-shaded
+[info] celeborn-common
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-spark-common
+[info] celeborn-spark-group
+[info] celeborn-spark-it
+[info] celeborn-worker
+```
+
+To enable the Flink 1.15 client module, add `-Pflink-1.15`:
+
+```
+# ./build/sbt -Pflink-1.15 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-flink-1_15
+[info] celeborn-client-flink-1_15-shaded
+[info] celeborn-common
+[info] celeborn-flink-common
+[info] celeborn-flink-group
+[info] celeborn-flink-it
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-worker
+```
+
+By using these profiles, you can easily switch between different client modules for Spark and Flink. These profiles enable specific dependencies and configurations relevant to the chosen version. This way, you can conveniently manage and build the desired configurations of the Celeborn project.
+
+## Building Spark/Flink Assembly Client Jars
+
+For example, you can build the Spark 3.3 client assembly jar by running the following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-spark-3-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pspark-3.3` profile:
+$ ./build/sbt -Pspark-3.3 celeborn-client-spark-3-shaded/assembly
+```
+
+Similarly, you can build the Flink 1.15 client assembly jar using the following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-flink-1_15-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pflink-1.15` profile:
+$ ./build/sbt -Pflink-1.15 celeborn-client-flink-1_15-shaded/assembly
+```
+
+By executing these commands, you will create assembly jar files for the respective Spark and Flink client modules. The assembly jar bundles all the dependencies, allowing the client module to be used independently with all required dependencies included.
+
+## Building submodules individually
+
+For instance, you can build the Celeborn Client module using:
+
+```
+$ # sbt
+$ build/sbt
+> project celeborn-client
+> package
+
+$ # Or, you can build the celeborn-client module with sbt directly using:
+$ build/sbt celeborn-client/package
+```
+
+## Testing with SBT
+
+To run all tests for the Celeborn project, you can use the following command:
+
+```shell
+./build/sbt test
+```
+
+Running tests for specific versions of Spark/Flink client.
+
+For example, to run the test cases for the Spark 3.3 client, use the following command:
+
+```shell
+$ ./build/sbt -Pspark-3.3 test
+
+$ # only run spark client related modules tests
+$ ./build/sbt -Pspark-3.3 celeborn-spark-group/test
+```
+
+Similarly, to run the test cases for the Flink 1.15 client, use the following command:
+
+```shell
+$ ./build/sbt -Pflink-1.15 test
+
+$ # only run flink client related modules tests
+$ ./build/sbt -Pflink-1.15 celeborn-flink-group/test
+```
+
+## Running Individual Tests
+
+When developing locally, it’s often convenient to run a single test or a few tests, rather than running the entire test suite.
+
+The fastest way to run individual tests is to use the sbt console. It’s fastest to keep a sbt console open, and use it to re-run tests as necessary. For example, to run all of the tests in a particular project, e.g., client:
+
+```
+$ build/sbt
+> project celeborn-client
+> test
+```
+You can run a single test suite using the `testOnly` command. For example, to run the `ShuffleClientSuiteJ`:
+
+```
+> testOnly org.apache.celeborn.client.ShuffleClientSuiteJ
+```
+The `testOnly` command accepts wildcards; e.g., you can also run the `ShuffleClientSuiteJ` with:
+
+```
+> testOnly *ShuffleClientSuiteJ
+```
+Or you could run all of the tests in the `client` package:
+
+```
+> testOnly org.apache.celeborn.client.*
+```
+If you’d like to run just a single Java test in the `ShuffleClientSuiteJ`, e.g., a test that with the name `testPushData`, you run the following command in the sbt console:
+
+```
+> testOnly *ShuffleClientSuiteJ -- *ShuffleClientSuiteJ.testPushData
+```
+
+If you’d like to run just a single Scala test in the `WorkerStatusTrackerSuite`, e.g., a test that incudes "handleHeartbeatResponse" in the name, you run the following command in the sbt console:
+
+```
+> testOnly *WorkerStatusTrackerSuite -- -z handleHeartbeatResponse
+```
+
+If you’d prefer, you can run all of these commands on the command line (but this will be slower than running tests using an open console). To do this, you need to surround `testOnly` and the following arguments in quotes:
+
+```
+$ build/sbt "celeborn-client/testOnly *WorkerStatusTrackerSuite -- -z handleHeartbeatResponse"
+```
+For more about how to run individual tests with sbt, see the [sbt documentation](https://www.scala-sbt.org/1.x/docs/Testing.html) and [JUnit Interface](https://github.com/sbt/junit-interface/#junit-interface).
+
+# Accelerating SBT
Review Comment:
Do you have any suggestions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 closed pull request #1795: [CELEBORN-877][DOC] Document on SBT
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed pull request #1795: [CELEBORN-877][DOC] Document on SBT
URL: https://github.com/apache/incubator-celeborn/pull/1795
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1795: [CELEBORN-877][DOC] Document on SBT
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1795:
URL: https://github.com/apache/incubator-celeborn/pull/1795#discussion_r1287692868
##########
docs/sbt.md:
##########
@@ -0,0 +1,275 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+
+# Building via SBT
+
+Starting from version 0.4.0, the Celeborn project supports building and packaging using SBT. This article provides a detailed guide on how to build the Celeborn project using SBT.
+
+# System Requirements
+
+Celeborn Service (master/worker) supports Scala 2.11/Scala 2.12 and Java 8/11/17.
+
+The table following indicates the compatibility of Celeborn Spark and Flink clients with different versions of Spark and Flink for various Java and Scala versions:
+
+| | Java 8/Scala 2.11 | Java 8/Scala 2.12 | Java 11/Scala 2.12 | Java 17/Scala 2.12 |
+|----------------|-------------------|-------------------|--------------------|--------------------|
+| Spark 2.4 | ✔ | ❌ | ❌ | ❌ |
+| Spark 3.0 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.1 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.2 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.3 | ❌ | ✔ | ✔ | ✔ |
+| Spark 3.4 | ❌ | ✔ | ✔ | ✔ |
+| Flink 1.14 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.15 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.17 | ❌ | ✔ | ✔ | ❌ |
+
+
+# Setting up Private Repository Proxy
+
+If you don't encounter any network issues, you can skip this step.
+
+## Setting up Repositories for Chinese Developers
+
+As a developer in China, you might face slow resource loading during the sbt startup process. To accelerate resource retrieval, you can replace the default repositories with the following configurations.
Review Comment:
not only china mainland, there are many areas that may face low speed issues, it's a generic technical issue. the thing we can do is to provide area-specific `repositories-<area>.template`s with verified mirror.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1795: [CELEBORN-877][DOC] Document on SBT
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1795:
URL: https://github.com/apache/incubator-celeborn/pull/1795#discussion_r1290839405
##########
docs/developers/sbt.md:
##########
@@ -0,0 +1,277 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Building via SBT
+
+Starting from version 0.4.0, the Celeborn project supports building and packaging using SBT. This article provides a detailed guide on how to build the Celeborn project using SBT.
+
+# System Requirements
+
+Celeborn Service (master/worker) supports Scala 2.11/2.12 and Java 8/11/17.
+
+The following table indicates the compatibility of Celeborn Spark and Flink clients with different versions of Spark and Flink for various Java and Scala versions:
+
+| | Java 8/Scala 2.11 | Java 8/Scala 2.12 | Java 11/Scala 2.12 | Java 17/Scala 2.12 |
+|----------------|-------------------|-------------------|--------------------|--------------------|
+| Spark 2.4 | ✔ | ❌ | ❌ | ❌ |
+| Spark 3.0 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.1 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.2 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.3 | ❌ | ✔ | ✔ | ✔ |
+| Spark 3.4 | ❌ | ✔ | ✔ | ✔ |
+| Flink 1.14 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.15 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.17 | ❌ | ✔ | ✔ | ❌ |
+
+# Useful SBT commands
+
+## Packaging the Project
+
+As an example, one can build a version of Celeborn as follows:
+
+```
+./build/sbt clean package
+```
+
+To create a Celeborn distribution like those distributed by the [Celeborn Downloads](https://celeborn.apache.org/download/) page, and that is laid out so as to be runnable, use `./build/make-distribution.sh` in the project root directory.
+
+```
+./build/make-distribution.sh --sbt
+```
+
+## Maven-Style Profile Management
+
+We have adopted the Maven-style profile management for our Client module. For example, you can enable the Spark 3.3 client module by adding `-Pspark-3.3`:
+
+```
+# ./build/sbt -Pspark-3.3 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-spark-3
+[info] celeborn-client-spark-3-shaded
+[info] celeborn-common
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-spark-common
+[info] celeborn-spark-group
+[info] celeborn-spark-it
+[info] celeborn-worker
+```
+
+To enable the Flink 1.15 client module, add `-Pflink-1.15`:
+
+```
+# ./build/sbt -Pflink-1.15 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-flink-1_15
+[info] celeborn-client-flink-1_15-shaded
+[info] celeborn-common
+[info] celeborn-flink-common
+[info] celeborn-flink-group
+[info] celeborn-flink-it
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-worker
+```
+
+By using these profiles, you can easily switch between different client modules for Spark and Flink. These profiles enable specific dependencies and configurations relevant to the chosen version. This way, you can conveniently manage and build the desired configurations of the Celeborn project.
+
+## Building Spark/Flink Assembly Client Jars
+
+For example, you can build the Spark 3.3 client assembly jar by running the following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-spark-3-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pspark-3.3` profile:
+$ ./build/sbt -Pspark-3.3 celeborn-client-spark-3-shaded/assembly
+```
+
+Similarly, you can build the Flink 1.15 client assembly jar using the following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-flink-1_15-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pflink-1.15` profile:
+$ ./build/sbt -Pflink-1.15 celeborn-client-flink-1_15-shaded/assembly
+```
+
+By executing these commands, you will create assembly jar files for the respective Spark and Flink client modules. The assembly jar bundles all the dependencies, allowing the client module to be used independently with all required dependencies included.
+
+## Building submodules individually
+
+For instance, you can build the Celeborn Client module using:
+
+```
+$ # sbt
+$ build/sbt
+> project celeborn-client
+> package
+
+$ # Or, you can build the celeborn-client module with sbt directly using:
+$ build/sbt celeborn-client/package
+```
+
+## Testing with SBT
+
+To run all tests for the Celeborn project, you can use the following command:
+
+```shell
+./build/sbt test
+```
+
+Running tests for specific versions of Spark/Flink client.
+
+For example, to run the test cases for the Spark 3.3 client, use the following command:
+
+```shell
+$ ./build/sbt -Pspark-3.3 test
+
+$ # only run spark client related modules tests
+$ ./build/sbt -Pspark-3.3 celeborn-spark-group/test
+```
+
+Similarly, to run the test cases for the Flink 1.15 client, use the following command:
+
+```shell
+$ ./build/sbt -Pflink-1.15 test
+
+$ # only run flink client related modules tests
+$ ./build/sbt -Pflink-1.15 celeborn-flink-group/test
+```
+
+## Running Individual Tests
+
+When developing locally, it’s often convenient to run a single test or a few tests, rather than running the entire test suite.
+
+The fastest way to run individual tests is to use the sbt console. It’s fastest to keep a sbt console open, and use it to re-run tests as necessary. For example, to run all of the tests in a particular project, e.g., client:
+
+```
+$ build/sbt
+> project celeborn-client
+> test
+```
+You can run a single test suite using the `testOnly` command. For example, to run the `ShuffleClientSuiteJ`:
+
+```
+> testOnly org.apache.celeborn.client.ShuffleClientSuiteJ
+```
+The `testOnly` command accepts wildcards; e.g., you can also run the `ShuffleClientSuiteJ` with:
+
+```
+> testOnly *ShuffleClientSuiteJ
+```
+Or you could run all of the tests in the `client` package:
+
+```
+> testOnly org.apache.celeborn.client.*
+```
+If you’d like to run just a single Java test in the `ShuffleClientSuiteJ`, e.g., a test that with the name `testPushData`, you run the following command in the sbt console:
+
+```
+> testOnly *ShuffleClientSuiteJ -- *ShuffleClientSuiteJ.testPushData
+```
+
+If you’d like to run just a single Scala test in the `WorkerStatusTrackerSuite`, e.g., a test that incudes "handleHeartbeatResponse" in the name, you run the following command in the sbt console:
+
+```
+> testOnly *WorkerStatusTrackerSuite -- -z handleHeartbeatResponse
Review Comment:
updated, and replace all examples from the `client` project to the `master` project
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1795: [CELEBORN-877][DOC] Document on SBT
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1795:
URL: https://github.com/apache/incubator-celeborn/pull/1795#discussion_r1290171009
##########
docs/developers/sbt.md:
##########
@@ -0,0 +1,276 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Building via SBT
+
+Starting from version 0.4.0, the Celeborn project supports building and packaging using SBT. This article provides a detailed guide on how to build the Celeborn project using SBT.
+
+# System Requirements
+
+Celeborn Service (master/worker) supports Scala 2.11/Scala 2.12 and Java 8/11/17.
+
+The table following indicates the compatibility of Celeborn Spark and Flink clients with different versions of Spark and Flink for various Java and Scala versions:
+
+| | Java 8/Scala 2.11 | Java 8/Scala 2.12 | Java 11/Scala 2.12 | Java 17/Scala 2.12 |
+|----------------|-------------------|-------------------|--------------------|--------------------|
+| Spark 2.4 | ✔ | ❌ | ❌ | ❌ |
+| Spark 3.0 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.1 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.2 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.3 | ❌ | ✔ | ✔ | ✔ |
+| Spark 3.4 | ❌ | ✔ | ✔ | ✔ |
+| Flink 1.14 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.15 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.17 | ❌ | ✔ | ✔ | ❌ |
+
+# Useful SBT commands
+
+## Packaging the Project
+
+As an example, one can build a version of Celeborn as follows:
+
+```
+./build/sbt clean package
+```
+
+To create a Celeborn distribution like those distributed by the [Celeborn Downloads](https://celeborn.apache.org/download/) page, and that is laid out so as to be runnable, use `./build/make-distribution.sh` in the project root directory.
+
+```
+./build/make-distribution.sh --sbt
+```
+
+## Maven-Style Profile Management
+
+We have adopted the Maven-style profile management for our Client module. For example, you can enable the Spark 3.3 client module by adding `-Pspark-3.3`:
+
+```
+# ./build/sbt -Pspark-3.3 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-spark-3
+[info] celeborn-client-spark-3-shaded
+[info] celeborn-common
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-spark-common
+[info] celeborn-spark-group
+[info] celeborn-spark-it
+[info] celeborn-worker
+```
+
+To enable the Flink 1.15 client module, add `-Pflink-1.15`:
+
+```
+# ./build/sbt -Pflink-1.15 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-flink-1_15
+[info] celeborn-client-flink-1_15-shaded
+[info] celeborn-common
+[info] celeborn-flink-common
+[info] celeborn-flink-group
+[info] celeborn-flink-it
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-worker
+```
+
+By using these profiles, you can easily switch between different client modules for Spark and Flink. These profiles enable specific dependencies and configurations relevant to the chosen version. This way, you can conveniently manage and build the desired configurations of the Celeborn project.
+
+## Building Spark/Flink Assembly Client Jars
+
+For example, you can build the Spark 3.3 client assembly jar by running the following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-spark-3-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pspark-3.3` profile:
+$ ./build/sbt -Pspark-3.3 celeborn-client-spark-3-shaded/assembly
+```
+
+Similarly, you can build the Flink 1.15 client assembly jar using the following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-flink-1_15-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pflink-1.15` profile:
+$ ./build/sbt -Pflink-1.15 celeborn-client-flink-1_15-shaded/assembly
+```
+
+By executing these commands, you will create assembly jar files for the respective Spark and Flink client modules. The assembly jar bundles all the dependencies, allowing the client module to be used independently with all required dependencies included.
+
+## Building submodules individually
+
+For instance, you can build the Celeborn Client module using:
+
+```
+$ # sbt
+$ build/sbt
+> project celeborn-client
+> package
+
+$ # Or, you can build the celeborn-client module with sbt directly using:
+$ build/sbt celeborn-client/package
+```
+
+## Testing with SBT
+
+To run all tests for the Celeborn project, you can use the following command:
+
+```shell
+./build/sbt test
+```
+
+Running tests for specific versions of Spark/Flink client.
+
+For example, to run the test cases for the Spark 3.3 client, use the following command:
+
+```shell
+$ ./build/sbt -Pspark-3.3 test
+
+$ # only run spark client related modules tests
+$ ./build/sbt -Pspark-3.3 celeborn-spark-group/test
+```
+
+Similarly, to run the test cases for the Flink 1.15 client, use the following command:
+
+```shell
+$ ./build/sbt -Pflink-1.15 test
+
+$ # only run flink client related modules tests
+$ ./build/sbt -Pflink-1.15 celeborn-flink-group/test
+```
+
+## Running Individual Tests
+
+When developing locally, it’s often convenient to run a single test or a few tests, rather than running the entire test suite.
+
+The fastest way to run individual tests is to use the sbt console. It’s fastest to keep a sbt console open, and use it to re-run tests as necessary. For example, to run all of the tests in a particular project, e.g., client:
+
+```
+$ build/sbt
+> project celeborn-client
+> test
+```
+You can run a single test suite using the `testOnly` command. For example, to run the `ShuffleClientSuiteJ`:
+
+```
+> testOnly org.apache.celeborn.client.ShuffleClientSuiteJ
+```
+The `testOnly` command accepts wildcards; e.g., you can also run the `ShuffleClientSuiteJ` with:
+
+```
+> testOnly *ShuffleClientSuiteJ
+```
+Or you could run all of the tests in the `client` package:
+
+```
+> testOnly org.apache.celeborn.client.*
+```
+If you’d like to run just a single Java test in the `ShuffleClientSuiteJ`, e.g., a test that with the name `testPushData`, you run the following command in the sbt console:
+
+```
+> testOnly *ShuffleClientSuiteJ -- *ShuffleClientSuiteJ.testPushData
+```
+
+If you’d like to run just a single Scala test in the `WorkerStatusTrackerSuite`, e.g., a test that incudes "handleHeartbeatResponse" in the name, you run the following command in the sbt console:
+
+```
+> testOnly *WorkerStatusTrackerSuite -- -z handleHeartbeatResponse
+```
+
+If you’d prefer, you can run all of these commands on the command line (but this will be slower than running tests using an open console). To do this, you need to surround `testOnly` and the following arguments in quotes:
+
+```
+$ build/sbt "celeborn-client/testOnly *WorkerStatusTrackerSuite -- -z handleHeartbeatResponse"
+```
+For more about how to run individual tests with sbt, see the [sbt documentation](https://www.scala-sbt.org/1.x/docs/Testing.html) and [JUnit Interface](https://github.com/sbt/junit-interface/#junit-interface).
+
+# Accelerating SBT
+
+This section provides instructions on setting up repository proxies for a smoother SBT experience. Depending on your location and network conditions, you can choose the appropriate approach to accelerate SBT startup and enhance dependency retrieval.
+
+## Accelerating SBT Startup
+
+The SBT startup process involves fetching the SBT bootstrap jar, which is typically obtained from the Maven Central Repository (https://repo1.maven.org/maven2/). If you encounter slow access to this repository or if it's inaccessible in your network environment, you can expedite the SBT startup by configuring a custom artifact repository using the `DEFAULT_ARTIFACT_REPOSITORY` environment variable.
+
+
+```shell
+$ # The following command fetches sbt-launch-x.y.z.jar from https://maven.aliyun.com/nexus/content/groups/public/
+$ # Ensure that the URL ends with a trailing slash "/"
+$ export DEFAULT_ARTIFACT_REPOSITORY=https://maven.aliyun.com/nexus/content/groups/public/ && ./build/sbt
+```
+
+This will initiate SBT using the specified repository, allowing for faster download and startup times.
+
+## Custom SBT Repositories
+
+The current repositories embedded within the Celeborn project are detailed below:
+
+```
+[repositories]
+ local
+ mavenLocal: file://${user.home}/.m2/repository/
+ local-preloaded-ivy: file:///${sbt.preloaded-${sbt.global.base-${user.home}/.sbt}/preloaded/}, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext]
+ local-preloaded: file:///${sbt.preloaded-${sbt.global.base-${user.home}/.sbt}/preloaded/}
+ # The system property value of `celeborn.sbt.default.artifact.repository` is
+ # fetched from the environment variable `DEFAULT_ARTIFACT_REPOSITORY` and
+ # assigned within the build/sbt-launch-lib.bash script.
+ private: ${celeborn.sbt.default.artifact.repository-file:///dev/null}
+ gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/repos/central/data/
+ maven-central
+ typesafe-ivy-releases: https://repo.typesafe.com/typesafe/ivy-releases/, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext], bootOnly
+ sbt-ivy-snapshots: https://repo.scala-sbt.org/scalasbt/ivy-snapshots/, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext], bootOnly
+ sbt-plugin-releases: https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
+ bintray-typesafe-sbt-plugin-releases: https://dl.bintray.com/typesafe/sbt-plugins/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
+ bintray-spark-packages: https://dl.bintray.com/spark-packages/maven/
+ typesafe-releases: https://repo.typesafe.com/typesafe/releases/
+```
+
+For numerous developers across various regions, the default repository download speeds are less than optimal. Consider, for example, developers situated in mainland China. In such cases, the following command can be employed to enhance dependency download speeds:
+
+```
+cp build/sbt-config/repositories-cn.template build/sbt-config/repositories-local
+```
+
+> NOTE: <br>
Review Comment:
yep, if there is no br, it will look like this:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1795: [CELEBORN-877][DOC] Document on SBT
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1795:
URL: https://github.com/apache/incubator-celeborn/pull/1795#discussion_r1290101840
##########
docs/cluster_planning.md:
##########
@@ -1,12 +1,14 @@
---
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
+ contributor license agreements. See the NOTICE file distributed with
Review Comment:
super nit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] codecov[bot] commented on pull request #1795: [CELEBORN-877][DOC] Document on SBT
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #1795:
URL: https://github.com/apache/incubator-celeborn/pull/1795#issuecomment-1667795125
## [Codecov](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1795?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#1795](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1795?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (555226c) into [main](https://app.codecov.io/gh/apache/incubator-celeborn/commit/6ea1ee2ec424c1011eef80a5d74c54cc1464f471?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (6ea1ee2) will **decrease** coverage by `0.01%`.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## main #1795 +/- ##
==========================================
- Coverage 46.60% 46.58% -0.01%
==========================================
Files 162 162
Lines 10079 10079
Branches 928 928
==========================================
- Hits 4696 4694 -2
- Misses 5073 5074 +1
- Partials 310 311 +1
```
[see 1 file with indirect coverage changes](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1795/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1795: [CELEBORN-877][DOC] Document on SBT
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1795:
URL: https://github.com/apache/incubator-celeborn/pull/1795#discussion_r1287690830
##########
docs/sbt.md:
##########
@@ -0,0 +1,275 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+
+# Building via SBT
+
+Starting from version 0.4.0, the Celeborn project supports building and packaging using SBT. This article provides a detailed guide on how to build the Celeborn project using SBT.
+
+# System Requirements
+
+Celeborn Service (master/worker) supports Scala 2.11/Scala 2.12 and Java 8/11/17.
+
+The table following indicates the compatibility of Celeborn Spark and Flink clients with different versions of Spark and Flink for various Java and Scala versions:
+
+| | Java 8/Scala 2.11 | Java 8/Scala 2.12 | Java 11/Scala 2.12 | Java 17/Scala 2.12 |
+|----------------|-------------------|-------------------|--------------------|--------------------|
+| Spark 2.4 | ✔ | ❌ | ❌ | ❌ |
+| Spark 3.0 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.1 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.2 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.3 | ❌ | ✔ | ✔ | ✔ |
+| Spark 3.4 | ❌ | ✔ | ✔ | ✔ |
+| Flink 1.14 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.15 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.17 | ❌ | ✔ | ✔ | ❌ |
+
+
+# Setting up Private Repository Proxy
+
+If you don't encounter any network issues, you can skip this step.
+
+## Setting up Repositories for Chinese Developers
+
+As a developer in China, you might face slow resource loading during the sbt startup process. To accelerate resource retrieval, you can replace the default repositories with the following configurations.
+
+Copy the following content into `${CELEBORN_HOME}/build/sbt-config/repositories`:
Review Comment:
it's not intend to guide user touch the content of `repositories`, instead, `repositories-local` should be used for any case that user want to change the repo stuff, e.g. use private repo, or mirror.
BTW, we can also mention that `repositories-local` is ignored by git, creating such file won't mess the codebase and git commit history.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1795: [CELEBORN-877][DOC] Document on SBT
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1795:
URL: https://github.com/apache/incubator-celeborn/pull/1795#discussion_r1290204019
##########
docs/developers/sbt.md:
##########
@@ -0,0 +1,276 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Building via SBT
+
+Starting from version 0.4.0, the Celeborn project supports building and packaging using SBT. This article provides a detailed guide on how to build the Celeborn project using SBT.
+
+# System Requirements
+
+Celeborn Service (master/worker) supports Scala 2.11/Scala 2.12 and Java 8/11/17.
+
+The table following indicates the compatibility of Celeborn Spark and Flink clients with different versions of Spark and Flink for various Java and Scala versions:
+
+| | Java 8/Scala 2.11 | Java 8/Scala 2.12 | Java 11/Scala 2.12 | Java 17/Scala 2.12 |
+|----------------|-------------------|-------------------|--------------------|--------------------|
+| Spark 2.4 | ✔ | ❌ | ❌ | ❌ |
+| Spark 3.0 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.1 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.2 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.3 | ❌ | ✔ | ✔ | ✔ |
+| Spark 3.4 | ❌ | ✔ | ✔ | ✔ |
+| Flink 1.14 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.15 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.17 | ❌ | ✔ | ✔ | ❌ |
+
+# Useful SBT commands
+
+## Packaging the Project
+
+As an example, one can build a version of Celeborn as follows:
+
+```
+./build/sbt clean package
+```
+
+To create a Celeborn distribution like those distributed by the [Celeborn Downloads](https://celeborn.apache.org/download/) page, and that is laid out so as to be runnable, use `./build/make-distribution.sh` in the project root directory.
+
+```
+./build/make-distribution.sh --sbt
Review Comment:
SGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1795: [CELEBORN-877][DOC] Document on SBT
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1795:
URL: https://github.com/apache/incubator-celeborn/pull/1795#discussion_r1290357010
##########
docs/developers/sbt.md:
##########
@@ -0,0 +1,277 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Building via SBT
+
+Starting from version 0.4.0, the Celeborn project supports building and packaging using SBT. This article provides a detailed guide on how to build the Celeborn project using SBT.
+
+# System Requirements
+
+Celeborn Service (master/worker) supports Scala 2.11/2.12 and Java 8/11/17.
+
+The following table indicates the compatibility of Celeborn Spark and Flink clients with different versions of Spark and Flink for various Java and Scala versions:
+
+| | Java 8/Scala 2.11 | Java 8/Scala 2.12 | Java 11/Scala 2.12 | Java 17/Scala 2.12 |
+|----------------|-------------------|-------------------|--------------------|--------------------|
+| Spark 2.4 | ✔ | ❌ | ❌ | ❌ |
+| Spark 3.0 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.1 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.2 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.3 | ❌ | ✔ | ✔ | ✔ |
+| Spark 3.4 | ❌ | ✔ | ✔ | ✔ |
+| Flink 1.14 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.15 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.17 | ❌ | ✔ | ✔ | ❌ |
+
+# Useful SBT commands
+
+## Packaging the Project
+
+As an example, one can build a version of Celeborn as follows:
+
+```
+./build/sbt clean package
+```
+
+To create a Celeborn distribution like those distributed by the [Celeborn Downloads](https://celeborn.apache.org/download/) page, and that is laid out so as to be runnable, use `./build/make-distribution.sh` in the project root directory.
+
+```
+./build/make-distribution.sh --sbt
+```
+
+## Maven-Style Profile Management
+
+We have adopted the Maven-style profile management for our Client module. For example, you can enable the Spark 3.3 client module by adding `-Pspark-3.3`:
+
+```
+# ./build/sbt -Pspark-3.3 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-spark-3
+[info] celeborn-client-spark-3-shaded
+[info] celeborn-common
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-spark-common
+[info] celeborn-spark-group
+[info] celeborn-spark-it
+[info] celeborn-worker
+```
+
+To enable the Flink 1.15 client module, add `-Pflink-1.15`:
+
+```
+# ./build/sbt -Pflink-1.15 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-flink-1_15
+[info] celeborn-client-flink-1_15-shaded
+[info] celeborn-common
+[info] celeborn-flink-common
+[info] celeborn-flink-group
+[info] celeborn-flink-it
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-worker
+```
+
+By using these profiles, you can easily switch between different client modules for Spark and Flink. These profiles enable specific dependencies and configurations relevant to the chosen version. This way, you can conveniently manage and build the desired configurations of the Celeborn project.
+
+## Building Spark/Flink Assembly Client Jars
+
+For example, you can build the Spark 3.3 client assembly jar by running the following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-spark-3-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pspark-3.3` profile:
+$ ./build/sbt -Pspark-3.3 celeborn-client-spark-3-shaded/assembly
+```
+
+Similarly, you can build the Flink 1.15 client assembly jar using the following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-flink-1_15-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pflink-1.15` profile:
+$ ./build/sbt -Pflink-1.15 celeborn-client-flink-1_15-shaded/assembly
+```
+
+By executing these commands, you will create assembly jar files for the respective Spark and Flink client modules. The assembly jar bundles all the dependencies, allowing the client module to be used independently with all required dependencies included.
+
+## Building submodules individually
+
+For instance, you can build the Celeborn Client module using:
+
+```
+$ # sbt
+$ build/sbt
+> project celeborn-client
+> package
+
+$ # Or, you can build the celeborn-client module with sbt directly using:
+$ build/sbt celeborn-client/package
+```
+
+## Testing with SBT
+
+To run all tests for the Celeborn project, you can use the following command:
+
+```shell
+./build/sbt test
+```
+
+Running tests for specific versions of Spark/Flink client.
+
+For example, to run the test cases for the Spark 3.3 client, use the following command:
+
+```shell
+$ ./build/sbt -Pspark-3.3 test
+
+$ # only run spark client related modules tests
+$ ./build/sbt -Pspark-3.3 celeborn-spark-group/test
+```
+
+Similarly, to run the test cases for the Flink 1.15 client, use the following command:
+
+```shell
+$ ./build/sbt -Pflink-1.15 test
+
+$ # only run flink client related modules tests
+$ ./build/sbt -Pflink-1.15 celeborn-flink-group/test
+```
+
+## Running Individual Tests
+
+When developing locally, it’s often convenient to run a single test or a few tests, rather than running the entire test suite.
+
+The fastest way to run individual tests is to use the sbt console. It’s fastest to keep a sbt console open, and use it to re-run tests as necessary. For example, to run all of the tests in a particular project, e.g., client:
+
+```
+$ build/sbt
+> project celeborn-client
+> test
+```
+You can run a single test suite using the `testOnly` command. For example, to run the `ShuffleClientSuiteJ`:
+
+```
+> testOnly org.apache.celeborn.client.ShuffleClientSuiteJ
+```
+The `testOnly` command accepts wildcards; e.g., you can also run the `ShuffleClientSuiteJ` with:
+
+```
+> testOnly *ShuffleClientSuiteJ
+```
+Or you could run all of the tests in the `client` package:
+
+```
+> testOnly org.apache.celeborn.client.*
+```
+If you’d like to run just a single Java test in the `ShuffleClientSuiteJ`, e.g., a test that with the name `testPushData`, you run the following command in the sbt console:
+
+```
+> testOnly *ShuffleClientSuiteJ -- *ShuffleClientSuiteJ.testPushData
+```
+
+If you’d like to run just a single Scala test in the `WorkerStatusTrackerSuite`, e.g., a test that incudes "handleHeartbeatResponse" in the name, you run the following command in the sbt console:
+
+```
+> testOnly *WorkerStatusTrackerSuite -- -z handleHeartbeatResponse
+```
+
+If you’d prefer, you can run all of these commands on the command line (but this will be slower than running tests using an open console). To do this, you need to surround `testOnly` and the following arguments in quotes:
+
+```
+$ build/sbt "celeborn-client/testOnly *WorkerStatusTrackerSuite -- -z handleHeartbeatResponse"
+```
+For more about how to run individual tests with sbt, see the [sbt documentation](https://www.scala-sbt.org/1.x/docs/Testing.html) and [JUnit Interface](https://github.com/sbt/junit-interface/#junit-interface).
+
+# Accelerating SBT
+
+This section provides instructions on setting up repository proxies for a smoother SBT experience. Depending on your location and network conditions, you can choose the appropriate approach to accelerate SBT startup and enhance dependency retrieval.
Review Comment:
proxies => proxies or mirrors
they are different
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1795: [CELEBORN-877][DOC] Document on SBT
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1795:
URL: https://github.com/apache/incubator-celeborn/pull/1795#discussion_r1290356370
##########
docs/developers/sbt.md:
##########
@@ -0,0 +1,277 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Building via SBT
+
+Starting from version 0.4.0, the Celeborn project supports building and packaging using SBT. This article provides a detailed guide on how to build the Celeborn project using SBT.
+
+# System Requirements
+
+Celeborn Service (master/worker) supports Scala 2.11/2.12 and Java 8/11/17.
+
+The following table indicates the compatibility of Celeborn Spark and Flink clients with different versions of Spark and Flink for various Java and Scala versions:
+
+| | Java 8/Scala 2.11 | Java 8/Scala 2.12 | Java 11/Scala 2.12 | Java 17/Scala 2.12 |
+|----------------|-------------------|-------------------|--------------------|--------------------|
+| Spark 2.4 | ✔ | ❌ | ❌ | ❌ |
+| Spark 3.0 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.1 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.2 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.3 | ❌ | ✔ | ✔ | ✔ |
+| Spark 3.4 | ❌ | ✔ | ✔ | ✔ |
+| Flink 1.14 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.15 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.17 | ❌ | ✔ | ✔ | ❌ |
+
+# Useful SBT commands
+
+## Packaging the Project
+
+As an example, one can build a version of Celeborn as follows:
+
+```
+./build/sbt clean package
+```
+
+To create a Celeborn distribution like those distributed by the [Celeborn Downloads](https://celeborn.apache.org/download/) page, and that is laid out so as to be runnable, use `./build/make-distribution.sh` in the project root directory.
+
+```
+./build/make-distribution.sh --sbt
+```
+
+## Maven-Style Profile Management
+
+We have adopted the Maven-style profile management for our Client module. For example, you can enable the Spark 3.3 client module by adding `-Pspark-3.3`:
+
+```
+# ./build/sbt -Pspark-3.3 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-spark-3
+[info] celeborn-client-spark-3-shaded
+[info] celeborn-common
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-spark-common
+[info] celeborn-spark-group
+[info] celeborn-spark-it
+[info] celeborn-worker
+```
+
+To enable the Flink 1.15 client module, add `-Pflink-1.15`:
+
+```
+# ./build/sbt -Pflink-1.15 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-flink-1_15
+[info] celeborn-client-flink-1_15-shaded
+[info] celeborn-common
+[info] celeborn-flink-common
+[info] celeborn-flink-group
+[info] celeborn-flink-it
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-worker
+```
+
+By using these profiles, you can easily switch between different client modules for Spark and Flink. These profiles enable specific dependencies and configurations relevant to the chosen version. This way, you can conveniently manage and build the desired configurations of the Celeborn project.
+
+## Building Spark/Flink Assembly Client Jars
+
+For example, you can build the Spark 3.3 client assembly jar by running the following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-spark-3-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pspark-3.3` profile:
+$ ./build/sbt -Pspark-3.3 celeborn-client-spark-3-shaded/assembly
+```
+
+Similarly, you can build the Flink 1.15 client assembly jar using the following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-flink-1_15-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pflink-1.15` profile:
+$ ./build/sbt -Pflink-1.15 celeborn-client-flink-1_15-shaded/assembly
+```
+
+By executing these commands, you will create assembly jar files for the respective Spark and Flink client modules. The assembly jar bundles all the dependencies, allowing the client module to be used independently with all required dependencies included.
+
+## Building submodules individually
+
+For instance, you can build the Celeborn Client module using:
+
+```
+$ # sbt
+$ build/sbt
+> project celeborn-client
+> package
+
+$ # Or, you can build the celeborn-client module with sbt directly using:
+$ build/sbt celeborn-client/package
+```
+
+## Testing with SBT
+
+To run all tests for the Celeborn project, you can use the following command:
+
+```shell
+./build/sbt test
+```
+
+Running tests for specific versions of Spark/Flink client.
+
+For example, to run the test cases for the Spark 3.3 client, use the following command:
+
+```shell
+$ ./build/sbt -Pspark-3.3 test
+
+$ # only run spark client related modules tests
+$ ./build/sbt -Pspark-3.3 celeborn-spark-group/test
+```
+
+Similarly, to run the test cases for the Flink 1.15 client, use the following command:
+
+```shell
+$ ./build/sbt -Pflink-1.15 test
+
+$ # only run flink client related modules tests
+$ ./build/sbt -Pflink-1.15 celeborn-flink-group/test
+```
+
+## Running Individual Tests
+
+When developing locally, it’s often convenient to run a single test or a few tests, rather than running the entire test suite.
+
+The fastest way to run individual tests is to use the sbt console. It’s fastest to keep a sbt console open, and use it to re-run tests as necessary. For example, to run all of the tests in a particular project, e.g., client:
+
+```
+$ build/sbt
+> project celeborn-client
+> test
+```
+You can run a single test suite using the `testOnly` command. For example, to run the `ShuffleClientSuiteJ`:
+
+```
+> testOnly org.apache.celeborn.client.ShuffleClientSuiteJ
+```
+The `testOnly` command accepts wildcards; e.g., you can also run the `ShuffleClientSuiteJ` with:
+
+```
+> testOnly *ShuffleClientSuiteJ
+```
+Or you could run all of the tests in the `client` package:
+
+```
+> testOnly org.apache.celeborn.client.*
+```
+If you’d like to run just a single Java test in the `ShuffleClientSuiteJ`, e.g., a test that with the name `testPushData`, you run the following command in the sbt console:
+
+```
+> testOnly *ShuffleClientSuiteJ -- *ShuffleClientSuiteJ.testPushData
+```
+
+If you’d like to run just a single Scala test in the `WorkerStatusTrackerSuite`, e.g., a test that incudes "handleHeartbeatResponse" in the name, you run the following command in the sbt console:
+
+```
+> testOnly *WorkerStatusTrackerSuite -- -z handleHeartbeatResponse
+```
+
+If you’d prefer, you can run all of these commands on the command line (but this will be slower than running tests using an open console). To do this, you need to surround `testOnly` and the following arguments in quotes:
+
+```
+$ build/sbt "celeborn-client/testOnly *WorkerStatusTrackerSuite -- -z handleHeartbeatResponse"
+```
+For more about how to run individual tests with sbt, see the [sbt documentation](https://www.scala-sbt.org/1.x/docs/Testing.html) and [JUnit Interface](https://github.com/sbt/junit-interface/#junit-interface).
+
+# Accelerating SBT
Review Comment:
too generic, it's network specific
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on pull request #1795: [CELEBORN-877][DOC] Document on SBT
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on PR #1795:
URL: https://github.com/apache/incubator-celeborn/pull/1795#issuecomment-1673174586
cc @pan3793 @waitinfuture PTAL when you have time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1795: [CELEBORN-877][DOC] Document on SBT
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1795:
URL: https://github.com/apache/incubator-celeborn/pull/1795#discussion_r1290106256
##########
docs/developers/sbt.md:
##########
@@ -0,0 +1,276 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Building via SBT
+
+Starting from version 0.4.0, the Celeborn project supports building and packaging using SBT. This article provides a detailed guide on how to build the Celeborn project using SBT.
+
+# System Requirements
+
+Celeborn Service (master/worker) supports Scala 2.11/Scala 2.12 and Java 8/11/17.
+
+The table following indicates the compatibility of Celeborn Spark and Flink clients with different versions of Spark and Flink for various Java and Scala versions:
+
+| | Java 8/Scala 2.11 | Java 8/Scala 2.12 | Java 11/Scala 2.12 | Java 17/Scala 2.12 |
+|----------------|-------------------|-------------------|--------------------|--------------------|
+| Spark 2.4 | ✔ | ❌ | ❌ | ❌ |
+| Spark 3.0 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.1 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.2 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.3 | ❌ | ✔ | ✔ | ✔ |
+| Spark 3.4 | ❌ | ✔ | ✔ | ✔ |
+| Flink 1.14 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.15 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.17 | ❌ | ✔ | ✔ | ❌ |
+
+# Useful SBT commands
+
+## Packaging the Project
+
+As an example, one can build a version of Celeborn as follows:
+
+```
+./build/sbt clean package
+```
+
+To create a Celeborn distribution like those distributed by the [Celeborn Downloads](https://celeborn.apache.org/download/) page, and that is laid out so as to be runnable, use `./build/make-distribution.sh` in the project root directory.
+
+```
+./build/make-distribution.sh --sbt
+```
+
+## Maven-Style Profile Management
+
+We have adopted the Maven-style profile management for our Client module. For example, you can enable the Spark 3.3 client module by adding `-Pspark-3.3`:
+
+```
+# ./build/sbt -Pspark-3.3 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-spark-3
+[info] celeborn-client-spark-3-shaded
+[info] celeborn-common
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-spark-common
+[info] celeborn-spark-group
+[info] celeborn-spark-it
+[info] celeborn-worker
+```
+
+To enable the Flink 1.15 client module, add `-Pflink-1.15`:
+
+```
+# ./build/sbt -Pflink-1.15 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-flink-1_15
+[info] celeborn-client-flink-1_15-shaded
+[info] celeborn-common
+[info] celeborn-flink-common
+[info] celeborn-flink-group
+[info] celeborn-flink-it
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-worker
+```
+
+By using these profiles, you can easily switch between different client modules for Spark and Flink. These profiles enable specific dependencies and configurations relevant to the chosen version. This way, you can conveniently manage and build the desired configurations of the Celeborn project.
+
+## Building Spark/Flink Assembly Client Jars
+
+For example, you can build the Spark 3.3 client assembly jar by running the following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-spark-3-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pspark-3.3` profile:
+$ ./build/sbt -Pspark-3.3 celeborn-client-spark-3-shaded/assembly
+```
+
+Similarly, you can build the Flink 1.15 client assembly jar using the following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-flink-1_15-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pflink-1.15` profile:
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1795: [CELEBORN-877][DOC] Document on SBT
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1795:
URL: https://github.com/apache/incubator-celeborn/pull/1795#discussion_r1287690830
##########
docs/sbt.md:
##########
@@ -0,0 +1,275 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+
+# Building via SBT
+
+Starting from version 0.4.0, the Celeborn project supports building and packaging using SBT. This article provides a detailed guide on how to build the Celeborn project using SBT.
+
+# System Requirements
+
+Celeborn Service (master/worker) supports Scala 2.11/Scala 2.12 and Java 8/11/17.
+
+The table following indicates the compatibility of Celeborn Spark and Flink clients with different versions of Spark and Flink for various Java and Scala versions:
+
+| | Java 8/Scala 2.11 | Java 8/Scala 2.12 | Java 11/Scala 2.12 | Java 17/Scala 2.12 |
+|----------------|-------------------|-------------------|--------------------|--------------------|
+| Spark 2.4 | ✔ | ❌ | ❌ | ❌ |
+| Spark 3.0 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.1 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.2 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.3 | ❌ | ✔ | ✔ | ✔ |
+| Spark 3.4 | ❌ | ✔ | ✔ | ✔ |
+| Flink 1.14 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.15 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.17 | ❌ | ✔ | ✔ | ❌ |
+
+
+# Setting up Private Repository Proxy
+
+If you don't encounter any network issues, you can skip this step.
+
+## Setting up Repositories for Chinese Developers
+
+As a developer in China, you might face slow resource loading during the sbt startup process. To accelerate resource retrieval, you can replace the default repositories with the following configurations.
+
+Copy the following content into `${CELEBORN_HOME}/build/sbt-config/repositories`:
Review Comment:
it's not intend to guide user touch the content of `repositories`, instead, `repositories-local` should be used for any case that user want to change the repo stuff, e.g. use private repo, or mirror
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on pull request #1795: [CELEBORN-877][DOC] Document on SBT
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #1795:
URL: https://github.com/apache/incubator-celeborn/pull/1795#issuecomment-1674193832
Thanks, merged to main
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1795: [CELEBORN-877][DOC] Document on SBT
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1795:
URL: https://github.com/apache/incubator-celeborn/pull/1795#discussion_r1290366969
##########
docs/developers/sbt.md:
##########
@@ -0,0 +1,277 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Building via SBT
+
+Starting from version 0.4.0, the Celeborn project supports building and packaging using SBT. This article provides a detailed guide on how to build the Celeborn project using SBT.
+
+# System Requirements
+
+Celeborn Service (master/worker) supports Scala 2.11/2.12 and Java 8/11/17.
+
+The following table indicates the compatibility of Celeborn Spark and Flink clients with different versions of Spark and Flink for various Java and Scala versions:
+
+| | Java 8/Scala 2.11 | Java 8/Scala 2.12 | Java 11/Scala 2.12 | Java 17/Scala 2.12 |
+|----------------|-------------------|-------------------|--------------------|--------------------|
+| Spark 2.4 | ✔ | ❌ | ❌ | ❌ |
+| Spark 3.0 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.1 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.2 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.3 | ❌ | ✔ | ✔ | ✔ |
+| Spark 3.4 | ❌ | ✔ | ✔ | ✔ |
+| Flink 1.14 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.15 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.17 | ❌ | ✔ | ✔ | ❌ |
+
+# Useful SBT commands
+
+## Packaging the Project
+
+As an example, one can build a version of Celeborn as follows:
+
+```
+./build/sbt clean package
+```
+
+To create a Celeborn distribution like those distributed by the [Celeborn Downloads](https://celeborn.apache.org/download/) page, and that is laid out so as to be runnable, use `./build/make-distribution.sh` in the project root directory.
+
+```
+./build/make-distribution.sh --sbt
+```
+
+## Maven-Style Profile Management
+
+We have adopted the Maven-style profile management for our Client module. For example, you can enable the Spark 3.3 client module by adding `-Pspark-3.3`:
+
+```
+# ./build/sbt -Pspark-3.3 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-spark-3
+[info] celeborn-client-spark-3-shaded
+[info] celeborn-common
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-spark-common
+[info] celeborn-spark-group
+[info] celeborn-spark-it
+[info] celeborn-worker
+```
+
+To enable the Flink 1.15 client module, add `-Pflink-1.15`:
+
+```
+# ./build/sbt -Pflink-1.15 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-flink-1_15
+[info] celeborn-client-flink-1_15-shaded
+[info] celeborn-common
+[info] celeborn-flink-common
+[info] celeborn-flink-group
+[info] celeborn-flink-it
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-worker
+```
+
+By using these profiles, you can easily switch between different client modules for Spark and Flink. These profiles enable specific dependencies and configurations relevant to the chosen version. This way, you can conveniently manage and build the desired configurations of the Celeborn project.
+
+## Building Spark/Flink Assembly Client Jars
+
+For example, you can build the Spark 3.3 client assembly jar by running the following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-spark-3-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pspark-3.3` profile:
+$ ./build/sbt -Pspark-3.3 celeborn-client-spark-3-shaded/assembly
+```
+
+Similarly, you can build the Flink 1.15 client assembly jar using the following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-flink-1_15-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pflink-1.15` profile:
+$ ./build/sbt -Pflink-1.15 celeborn-client-flink-1_15-shaded/assembly
+```
+
+By executing these commands, you will create assembly jar files for the respective Spark and Flink client modules. The assembly jar bundles all the dependencies, allowing the client module to be used independently with all required dependencies included.
+
+## Building submodules individually
+
+For instance, you can build the Celeborn Client module using:
+
+```
+$ # sbt
+$ build/sbt
+> project celeborn-client
+> package
+
+$ # Or, you can build the celeborn-client module with sbt directly using:
+$ build/sbt celeborn-client/package
+```
+
+## Testing with SBT
+
+To run all tests for the Celeborn project, you can use the following command:
+
+```shell
+./build/sbt test
+```
+
+Running tests for specific versions of Spark/Flink client.
+
+For example, to run the test cases for the Spark 3.3 client, use the following command:
+
+```shell
+$ ./build/sbt -Pspark-3.3 test
+
+$ # only run spark client related modules tests
+$ ./build/sbt -Pspark-3.3 celeborn-spark-group/test
+```
+
+Similarly, to run the test cases for the Flink 1.15 client, use the following command:
+
+```shell
+$ ./build/sbt -Pflink-1.15 test
+
+$ # only run flink client related modules tests
+$ ./build/sbt -Pflink-1.15 celeborn-flink-group/test
+```
+
+## Running Individual Tests
+
+When developing locally, it’s often convenient to run a single test or a few tests, rather than running the entire test suite.
+
+The fastest way to run individual tests is to use the sbt console. It’s fastest to keep a sbt console open, and use it to re-run tests as necessary. For example, to run all of the tests in a particular project, e.g., client:
+
+```
+$ build/sbt
+> project celeborn-client
+> test
+```
+You can run a single test suite using the `testOnly` command. For example, to run the `ShuffleClientSuiteJ`:
+
+```
+> testOnly org.apache.celeborn.client.ShuffleClientSuiteJ
+```
+The `testOnly` command accepts wildcards; e.g., you can also run the `ShuffleClientSuiteJ` with:
+
+```
+> testOnly *ShuffleClientSuiteJ
+```
+Or you could run all of the tests in the `client` package:
+
+```
+> testOnly org.apache.celeborn.client.*
+```
+If you’d like to run just a single Java test in the `ShuffleClientSuiteJ`, e.g., a test that with the name `testPushData`, you run the following command in the sbt console:
+
+```
+> testOnly *ShuffleClientSuiteJ -- *ShuffleClientSuiteJ.testPushData
+```
+
+If you’d like to run just a single Scala test in the `WorkerStatusTrackerSuite`, e.g., a test that incudes "handleHeartbeatResponse" in the name, you run the following command in the sbt console:
+
+```
+> testOnly *WorkerStatusTrackerSuite -- -z handleHeartbeatResponse
+```
+
+If you’d prefer, you can run all of these commands on the command line (but this will be slower than running tests using an open console). To do this, you need to surround `testOnly` and the following arguments in quotes:
+
+```
+$ build/sbt "celeborn-client/testOnly *WorkerStatusTrackerSuite -- -z handleHeartbeatResponse"
+```
+For more about how to run individual tests with sbt, see the [sbt documentation](https://www.scala-sbt.org/1.x/docs/Testing.html) and [JUnit Interface](https://github.com/sbt/junit-interface/#junit-interface).
+
+# Accelerating SBT
+
+This section provides instructions on setting up repository proxies for a smoother SBT experience. Depending on your location and network conditions, you can choose the appropriate approach to accelerate SBT startup and enhance dependency retrieval.
+
+## Accelerating SBT Startup
+
+The SBT startup process involves fetching the SBT bootstrap jar, which is typically obtained from the Maven Central Repository (https://repo1.maven.org/maven2/). If you encounter slow access to this repository or if it's inaccessible in your network environment, you can expedite the SBT startup by configuring a custom artifact repository using the `DEFAULT_ARTIFACT_REPOSITORY` environment variable.
+
+
+```shell
+$ # The following command fetches sbt-launch-x.y.z.jar from https://maven.aliyun.com/nexus/content/groups/public/
+$ # Ensure that the URL ends with a trailing slash "/"
+$ export DEFAULT_ARTIFACT_REPOSITORY=https://maven.aliyun.com/nexus/content/groups/public/
+$ ./build/sbt
+```
+
+This will initiate SBT using the specified repository, allowing for faster download and startup times.
+
+## Custom SBT Repositories
+
+The current repositories embedded within the Celeborn project are detailed below:
+
+```
+[repositories]
+ local
+ mavenLocal: file://${user.home}/.m2/repository/
+ local-preloaded-ivy: file:///${sbt.preloaded-${sbt.global.base-${user.home}/.sbt}/preloaded/}, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext]
+ local-preloaded: file:///${sbt.preloaded-${sbt.global.base-${user.home}/.sbt}/preloaded/}
+ # The system property value of `celeborn.sbt.default.artifact.repository` is
+ # fetched from the environment variable `DEFAULT_ARTIFACT_REPOSITORY` and
+ # assigned within the build/sbt-launch-lib.bash script.
+ private: ${celeborn.sbt.default.artifact.repository-file:///dev/null}
+ gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/repos/central/data/
+ maven-central
+ typesafe-ivy-releases: https://repo.typesafe.com/typesafe/ivy-releases/, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext], bootOnly
+ sbt-ivy-snapshots: https://repo.scala-sbt.org/scalasbt/ivy-snapshots/, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext], bootOnly
+ sbt-plugin-releases: https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
+ bintray-typesafe-sbt-plugin-releases: https://dl.bintray.com/typesafe/sbt-plugins/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
+ bintray-spark-packages: https://dl.bintray.com/spark-packages/maven/
+ typesafe-releases: https://repo.typesafe.com/typesafe/releases/
+```
+
+For numerous developers across various regions, the default repository download speeds are less than optimal. Consider, for example, developers situated in China mainland. In such cases, the following command can be employed to enhance dependency download speeds:
Review Comment:
We can mention that we provide some templates with verified faster public mirrors for regions which have local developores asked for. For example ...
And it's encouraged that developer from different regions to contribute templates for themselfves region
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1795: [CELEBORN-877][DOC] Document on SBT
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1795:
URL: https://github.com/apache/incubator-celeborn/pull/1795#discussion_r1290357010
##########
docs/developers/sbt.md:
##########
@@ -0,0 +1,277 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Building via SBT
+
+Starting from version 0.4.0, the Celeborn project supports building and packaging using SBT. This article provides a detailed guide on how to build the Celeborn project using SBT.
+
+# System Requirements
+
+Celeborn Service (master/worker) supports Scala 2.11/2.12 and Java 8/11/17.
+
+The following table indicates the compatibility of Celeborn Spark and Flink clients with different versions of Spark and Flink for various Java and Scala versions:
+
+| | Java 8/Scala 2.11 | Java 8/Scala 2.12 | Java 11/Scala 2.12 | Java 17/Scala 2.12 |
+|----------------|-------------------|-------------------|--------------------|--------------------|
+| Spark 2.4 | ✔ | ❌ | ❌ | ❌ |
+| Spark 3.0 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.1 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.2 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.3 | ❌ | ✔ | ✔ | ✔ |
+| Spark 3.4 | ❌ | ✔ | ✔ | ✔ |
+| Flink 1.14 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.15 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.17 | ❌ | ✔ | ✔ | ❌ |
+
+# Useful SBT commands
+
+## Packaging the Project
+
+As an example, one can build a version of Celeborn as follows:
+
+```
+./build/sbt clean package
+```
+
+To create a Celeborn distribution like those distributed by the [Celeborn Downloads](https://celeborn.apache.org/download/) page, and that is laid out so as to be runnable, use `./build/make-distribution.sh` in the project root directory.
+
+```
+./build/make-distribution.sh --sbt
+```
+
+## Maven-Style Profile Management
+
+We have adopted the Maven-style profile management for our Client module. For example, you can enable the Spark 3.3 client module by adding `-Pspark-3.3`:
+
+```
+# ./build/sbt -Pspark-3.3 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-spark-3
+[info] celeborn-client-spark-3-shaded
+[info] celeborn-common
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-spark-common
+[info] celeborn-spark-group
+[info] celeborn-spark-it
+[info] celeborn-worker
+```
+
+To enable the Flink 1.15 client module, add `-Pflink-1.15`:
+
+```
+# ./build/sbt -Pflink-1.15 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-flink-1_15
+[info] celeborn-client-flink-1_15-shaded
+[info] celeborn-common
+[info] celeborn-flink-common
+[info] celeborn-flink-group
+[info] celeborn-flink-it
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-worker
+```
+
+By using these profiles, you can easily switch between different client modules for Spark and Flink. These profiles enable specific dependencies and configurations relevant to the chosen version. This way, you can conveniently manage and build the desired configurations of the Celeborn project.
+
+## Building Spark/Flink Assembly Client Jars
+
+For example, you can build the Spark 3.3 client assembly jar by running the following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-spark-3-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pspark-3.3` profile:
+$ ./build/sbt -Pspark-3.3 celeborn-client-spark-3-shaded/assembly
+```
+
+Similarly, you can build the Flink 1.15 client assembly jar using the following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-flink-1_15-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pflink-1.15` profile:
+$ ./build/sbt -Pflink-1.15 celeborn-client-flink-1_15-shaded/assembly
+```
+
+By executing these commands, you will create assembly jar files for the respective Spark and Flink client modules. The assembly jar bundles all the dependencies, allowing the client module to be used independently with all required dependencies included.
+
+## Building submodules individually
+
+For instance, you can build the Celeborn Client module using:
+
+```
+$ # sbt
+$ build/sbt
+> project celeborn-client
+> package
+
+$ # Or, you can build the celeborn-client module with sbt directly using:
+$ build/sbt celeborn-client/package
+```
+
+## Testing with SBT
+
+To run all tests for the Celeborn project, you can use the following command:
+
+```shell
+./build/sbt test
+```
+
+Running tests for specific versions of Spark/Flink client.
+
+For example, to run the test cases for the Spark 3.3 client, use the following command:
+
+```shell
+$ ./build/sbt -Pspark-3.3 test
+
+$ # only run spark client related modules tests
+$ ./build/sbt -Pspark-3.3 celeborn-spark-group/test
+```
+
+Similarly, to run the test cases for the Flink 1.15 client, use the following command:
+
+```shell
+$ ./build/sbt -Pflink-1.15 test
+
+$ # only run flink client related modules tests
+$ ./build/sbt -Pflink-1.15 celeborn-flink-group/test
+```
+
+## Running Individual Tests
+
+When developing locally, it’s often convenient to run a single test or a few tests, rather than running the entire test suite.
+
+The fastest way to run individual tests is to use the sbt console. It’s fastest to keep a sbt console open, and use it to re-run tests as necessary. For example, to run all of the tests in a particular project, e.g., client:
+
+```
+$ build/sbt
+> project celeborn-client
+> test
+```
+You can run a single test suite using the `testOnly` command. For example, to run the `ShuffleClientSuiteJ`:
+
+```
+> testOnly org.apache.celeborn.client.ShuffleClientSuiteJ
+```
+The `testOnly` command accepts wildcards; e.g., you can also run the `ShuffleClientSuiteJ` with:
+
+```
+> testOnly *ShuffleClientSuiteJ
+```
+Or you could run all of the tests in the `client` package:
+
+```
+> testOnly org.apache.celeborn.client.*
+```
+If you’d like to run just a single Java test in the `ShuffleClientSuiteJ`, e.g., a test that with the name `testPushData`, you run the following command in the sbt console:
+
+```
+> testOnly *ShuffleClientSuiteJ -- *ShuffleClientSuiteJ.testPushData
+```
+
+If you’d like to run just a single Scala test in the `WorkerStatusTrackerSuite`, e.g., a test that incudes "handleHeartbeatResponse" in the name, you run the following command in the sbt console:
+
+```
+> testOnly *WorkerStatusTrackerSuite -- -z handleHeartbeatResponse
+```
+
+If you’d prefer, you can run all of these commands on the command line (but this will be slower than running tests using an open console). To do this, you need to surround `testOnly` and the following arguments in quotes:
+
+```
+$ build/sbt "celeborn-client/testOnly *WorkerStatusTrackerSuite -- -z handleHeartbeatResponse"
+```
+For more about how to run individual tests with sbt, see the [sbt documentation](https://www.scala-sbt.org/1.x/docs/Testing.html) and [JUnit Interface](https://github.com/sbt/junit-interface/#junit-interface).
+
+# Accelerating SBT
+
+This section provides instructions on setting up repository proxies for a smoother SBT experience. Depending on your location and network conditions, you can choose the appropriate approach to accelerate SBT startup and enhance dependency retrieval.
Review Comment:
proxies => mirrors or proxies
they are different
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1795: [CELEBORN-877][DOC] Document on SBT
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1795:
URL: https://github.com/apache/incubator-celeborn/pull/1795#discussion_r1290105589
##########
docs/developers/sbt.md:
##########
@@ -0,0 +1,276 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Building via SBT
+
+Starting from version 0.4.0, the Celeborn project supports building and packaging using SBT. This article provides a detailed guide on how to build the Celeborn project using SBT.
+
+# System Requirements
+
+Celeborn Service (master/worker) supports Scala 2.11/Scala 2.12 and Java 8/11/17.
+
+The table following indicates the compatibility of Celeborn Spark and Flink clients with different versions of Spark and Flink for various Java and Scala versions:
+
+| | Java 8/Scala 2.11 | Java 8/Scala 2.12 | Java 11/Scala 2.12 | Java 17/Scala 2.12 |
+|----------------|-------------------|-------------------|--------------------|--------------------|
+| Spark 2.4 | ✔ | ❌ | ❌ | ❌ |
+| Spark 3.0 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.1 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.2 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.3 | ❌ | ✔ | ✔ | ✔ |
+| Spark 3.4 | ❌ | ✔ | ✔ | ✔ |
+| Flink 1.14 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.15 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.17 | ❌ | ✔ | ✔ | ❌ |
+
+# Useful SBT commands
+
+## Packaging the Project
+
+As an example, one can build a version of Celeborn as follows:
+
+```
+./build/sbt clean package
+```
+
+To create a Celeborn distribution like those distributed by the [Celeborn Downloads](https://celeborn.apache.org/download/) page, and that is laid out so as to be runnable, use `./build/make-distribution.sh` in the project root directory.
+
+```
+./build/make-distribution.sh --sbt
+```
+
+## Maven-Style Profile Management
+
+We have adopted the Maven-style profile management for our Client module. For example, you can enable the Spark 3.3 client module by adding `-Pspark-3.3`:
+
+```
+# ./build/sbt -Pspark-3.3 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-spark-3
+[info] celeborn-client-spark-3-shaded
+[info] celeborn-common
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-spark-common
+[info] celeborn-spark-group
+[info] celeborn-spark-it
+[info] celeborn-worker
+```
+
+To enable the Flink 1.15 client module, add `-Pflink-1.15`:
+
+```
+# ./build/sbt -Pflink-1.15 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-flink-1_15
+[info] celeborn-client-flink-1_15-shaded
+[info] celeborn-common
+[info] celeborn-flink-common
+[info] celeborn-flink-group
+[info] celeborn-flink-it
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-worker
+```
+
+By using these profiles, you can easily switch between different client modules for Spark and Flink. These profiles enable specific dependencies and configurations relevant to the chosen version. This way, you can conveniently manage and build the desired configurations of the Celeborn project.
+
+## Building Spark/Flink Assembly Client Jars
+
+For example, you can build the Spark 3.3 client assembly jar by running the following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-spark-3-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pspark-3.3` profile:
Review Comment:
break it
##########
docs/developers/sbt.md:
##########
@@ -0,0 +1,276 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Building via SBT
+
+Starting from version 0.4.0, the Celeborn project supports building and packaging using SBT. This article provides a detailed guide on how to build the Celeborn project using SBT.
+
+# System Requirements
+
+Celeborn Service (master/worker) supports Scala 2.11/Scala 2.12 and Java 8/11/17.
+
+The table following indicates the compatibility of Celeborn Spark and Flink clients with different versions of Spark and Flink for various Java and Scala versions:
+
+| | Java 8/Scala 2.11 | Java 8/Scala 2.12 | Java 11/Scala 2.12 | Java 17/Scala 2.12 |
+|----------------|-------------------|-------------------|--------------------|--------------------|
+| Spark 2.4 | ✔ | ❌ | ❌ | ❌ |
+| Spark 3.0 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.1 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.2 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.3 | ❌ | ✔ | ✔ | ✔ |
+| Spark 3.4 | ❌ | ✔ | ✔ | ✔ |
+| Flink 1.14 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.15 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.17 | ❌ | ✔ | ✔ | ❌ |
+
+# Useful SBT commands
+
+## Packaging the Project
+
+As an example, one can build a version of Celeborn as follows:
+
+```
+./build/sbt clean package
+```
+
+To create a Celeborn distribution like those distributed by the [Celeborn Downloads](https://celeborn.apache.org/download/) page, and that is laid out so as to be runnable, use `./build/make-distribution.sh` in the project root directory.
+
+```
+./build/make-distribution.sh --sbt
+```
+
+## Maven-Style Profile Management
+
+We have adopted the Maven-style profile management for our Client module. For example, you can enable the Spark 3.3 client module by adding `-Pspark-3.3`:
+
+```
+# ./build/sbt -Pspark-3.3 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-spark-3
+[info] celeborn-client-spark-3-shaded
+[info] celeborn-common
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-spark-common
+[info] celeborn-spark-group
+[info] celeborn-spark-it
+[info] celeborn-worker
+```
+
+To enable the Flink 1.15 client module, add `-Pflink-1.15`:
+
+```
+# ./build/sbt -Pflink-1.15 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-flink-1_15
+[info] celeborn-client-flink-1_15-shaded
+[info] celeborn-common
+[info] celeborn-flink-common
+[info] celeborn-flink-group
+[info] celeborn-flink-it
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-worker
+```
+
+By using these profiles, you can easily switch between different client modules for Spark and Flink. These profiles enable specific dependencies and configurations relevant to the chosen version. This way, you can conveniently manage and build the desired configurations of the Celeborn project.
+
+## Building Spark/Flink Assembly Client Jars
+
+For example, you can build the Spark 3.3 client assembly jar by running the following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-spark-3-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pspark-3.3` profile:
Review Comment:
break it down
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] cfmcgrady commented on a diff in pull request #1795: [CELEBORN-877][DOC] Document on SBT
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #1795:
URL: https://github.com/apache/incubator-celeborn/pull/1795#discussion_r1290169332
##########
docs/developers/sbt.md:
##########
@@ -0,0 +1,276 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Building via SBT
+
+Starting from version 0.4.0, the Celeborn project supports building and packaging using SBT. This article provides a detailed guide on how to build the Celeborn project using SBT.
+
+# System Requirements
+
+Celeborn Service (master/worker) supports Scala 2.11/Scala 2.12 and Java 8/11/17.
+
+The table following indicates the compatibility of Celeborn Spark and Flink clients with different versions of Spark and Flink for various Java and Scala versions:
+
+| | Java 8/Scala 2.11 | Java 8/Scala 2.12 | Java 11/Scala 2.12 | Java 17/Scala 2.12 |
+|----------------|-------------------|-------------------|--------------------|--------------------|
+| Spark 2.4 | ✔ | ❌ | ❌ | ❌ |
+| Spark 3.0 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.1 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.2 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.3 | ❌ | ✔ | ✔ | ✔ |
+| Spark 3.4 | ❌ | ✔ | ✔ | ✔ |
+| Flink 1.14 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.15 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.17 | ❌ | ✔ | ✔ | ❌ |
+
+# Useful SBT commands
+
+## Packaging the Project
+
+As an example, one can build a version of Celeborn as follows:
+
+```
+./build/sbt clean package
+```
+
+To create a Celeborn distribution like those distributed by the [Celeborn Downloads](https://celeborn.apache.org/download/) page, and that is laid out so as to be runnable, use `./build/make-distribution.sh` in the project root directory.
+
+```
+./build/make-distribution.sh --sbt
Review Comment:
no, it's a placeholder now, I intend to file a separate PR to support this feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1795: [CELEBORN-877][DOC] Document on SBT
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1795:
URL: https://github.com/apache/incubator-celeborn/pull/1795#discussion_r1290171117
##########
docs/developers/sbt.md:
##########
@@ -0,0 +1,276 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Building via SBT
+
+Starting from version 0.4.0, the Celeborn project supports building and packaging using SBT. This article provides a detailed guide on how to build the Celeborn project using SBT.
+
+# System Requirements
+
+Celeborn Service (master/worker) supports Scala 2.11/Scala 2.12 and Java 8/11/17.
+
+The table following indicates the compatibility of Celeborn Spark and Flink clients with different versions of Spark and Flink for various Java and Scala versions:
+
+| | Java 8/Scala 2.11 | Java 8/Scala 2.12 | Java 11/Scala 2.12 | Java 17/Scala 2.12 |
+|----------------|-------------------|-------------------|--------------------|--------------------|
+| Spark 2.4 | ✔ | ❌ | ❌ | ❌ |
+| Spark 3.0 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.1 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.2 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.3 | ❌ | ✔ | ✔ | ✔ |
+| Spark 3.4 | ❌ | ✔ | ✔ | ✔ |
+| Flink 1.14 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.15 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.17 | ❌ | ✔ | ✔ | ❌ |
+
+# Useful SBT commands
+
+## Packaging the Project
+
+As an example, one can build a version of Celeborn as follows:
+
+```
+./build/sbt clean package
+```
+
+To create a Celeborn distribution like those distributed by the [Celeborn Downloads](https://celeborn.apache.org/download/) page, and that is laid out so as to be runnable, use `./build/make-distribution.sh` in the project root directory.
+
+```
+./build/make-distribution.sh --sbt
Review Comment:
but there is a `--mvn </path/mvn>`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1795: [CELEBORN-877][DOC] Document on SBT
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1795:
URL: https://github.com/apache/incubator-celeborn/pull/1795#discussion_r1290104522
##########
docs/developers/sbt.md:
##########
@@ -0,0 +1,276 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Building via SBT
+
+Starting from version 0.4.0, the Celeborn project supports building and packaging using SBT. This article provides a detailed guide on how to build the Celeborn project using SBT.
+
+# System Requirements
+
+Celeborn Service (master/worker) supports Scala 2.11/Scala 2.12 and Java 8/11/17.
+
+The table following indicates the compatibility of Celeborn Spark and Flink clients with different versions of Spark and Flink for various Java and Scala versions:
+
+| | Java 8/Scala 2.11 | Java 8/Scala 2.12 | Java 11/Scala 2.12 | Java 17/Scala 2.12 |
+|----------------|-------------------|-------------------|--------------------|--------------------|
+| Spark 2.4 | ✔ | ❌ | ❌ | ❌ |
+| Spark 3.0 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.1 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.2 | ❌ | ✔ | ✔ | ❌ |
+| Spark 3.3 | ❌ | ✔ | ✔ | ✔ |
+| Spark 3.4 | ❌ | ✔ | ✔ | ✔ |
+| Flink 1.14 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.15 | ❌ | ✔ | ✔ | ❌ |
+| Flink 1.17 | ❌ | ✔ | ✔ | ❌ |
+
+# Useful SBT commands
+
+## Packaging the Project
+
+As an example, one can build a version of Celeborn as follows:
+
+```
+./build/sbt clean package
+```
+
+To create a Celeborn distribution like those distributed by the [Celeborn Downloads](https://celeborn.apache.org/download/) page, and that is laid out so as to be runnable, use `./build/make-distribution.sh` in the project root directory.
+
+```
+./build/make-distribution.sh --sbt
Review Comment:
have we supported it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org