You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "kerwin-zk (via GitHub)" <gi...@apache.org> on 2023/08/02 03:45:43 UTC

[GitHub] [incubator-celeborn] kerwin-zk opened a new pull request, #1784: [CELEBORN-870][DOC] Document on usage together with Gluten

kerwin-zk opened a new pull request, #1784:
URL: https://github.com/apache/incubator-celeborn/pull/1784

   ### What changes were proposed in this pull request?
   As title.
   
   ### Why are the changes needed?
   As title.
   
   ### Does this PR introduce _any_ user-facing change?
   No.
   
   ### How was this patch tested?
   Manual test.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1784: [CELEBORN-870][DOC] Document on usage together with Gluten

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1784:
URL: https://github.com/apache/incubator-celeborn/pull/1784#discussion_r1281357708


##########
docs/developers/glutensupport.md:
##########
@@ -0,0 +1,51 @@
+---
+license: |
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+  http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+---
+
+# Gluten Support
+## Velox Backend
+
+[Gluten](https://github.com/oap-project/gluten) with velox backend supports Celeborn as remote shuffle service. Below introduction is used to enable this feature
+
+First refer to this URL(https://github.com/oap-project/gluten/blob/main/docs/get-started/Velox.md) to build Gluten with velox backend.
+
+When compiling the Gluten java module, you need to add rss, as follows:
+
+```
+mvn clean package -Pbackends-velox -Pspark-3.3 -Prss -DskipTests
+```
+
+Then add the paths of Gluten and Spark Celeborn Client packages to class path.
+
+Currently to use Gluten following configurations are required in spark-defaults.conf

Review Comment:
   ```suggestion
   Currently to use Gluten following configurations are required in `spark-defaults.conf`
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] pan3793 commented on pull request #1784: [CELEBORN-870][DOC] Document on usage together with Gluten

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #1784:
URL: https://github.com/apache/incubator-celeborn/pull/1784#issuecomment-1661451122

   it's better to document the version matrix too.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1784: [CELEBORN-870][DOC] Document on usage together with Gluten

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1784:
URL: https://github.com/apache/incubator-celeborn/pull/1784#discussion_r1281359164


##########
docs/developers/glutensupport.md:
##########
@@ -0,0 +1,51 @@
+---
+license: |
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+  http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+---
+
+# Gluten Support
+## Velox Backend
+
+[Gluten](https://github.com/oap-project/gluten) with velox backend supports Celeborn as remote shuffle service. Below introduction is used to enable this feature
+
+First refer to this URL(https://github.com/oap-project/gluten/blob/main/docs/get-started/Velox.md) to build Gluten with velox backend.
+
+When compiling the Gluten java module, you need to add rss, as follows:
+
+```
+mvn clean package -Pbackends-velox -Pspark-3.3 -Prss -DskipTests
+```
+
+Then add the paths of Gluten and Spark Celeborn Client packages to class path.

Review Comment:
   ```suggestion
   Then add the Gluten and Spark Celeborn Client packages to your Spark application's classpath(usually add them into `$SPARK_HOME/jars`).
   
   - Celeborn: xxxx.jar
   - Gluten: xxx.jar
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1784: [CELEBORN-870][DOC] Document on usage together with Gluten

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1784:
URL: https://github.com/apache/incubator-celeborn/pull/1784#discussion_r1281356150


##########
docs/developers/glutensupport.md:
##########
@@ -0,0 +1,51 @@
+---
+license: |
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+  http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+---
+
+# Gluten Support
+## Velox Backend
+
+[Gluten](https://github.com/oap-project/gluten) with velox backend supports Celeborn as remote shuffle service. Below introduction is used to enable this feature
+
+First refer to this URL(https://github.com/oap-project/gluten/blob/main/docs/get-started/Velox.md) to build Gluten with velox backend.
+
+When compiling the Gluten java module, you need to add rss, as follows:

Review Comment:
   ```suggestion
   When compiling the Gluten Java module, it's required to enable `rss` profile, as follows:
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] kerwin-zk commented on a diff in pull request #1784: [CELEBORN-870][DOC] Document on usage together with Gluten

Posted by "kerwin-zk (via GitHub)" <gi...@apache.org>.
kerwin-zk commented on code in PR #1784:
URL: https://github.com/apache/incubator-celeborn/pull/1784#discussion_r1281365316


##########
docs/developers/glutensupport.md:
##########
@@ -0,0 +1,51 @@
+---
+license: |
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+  http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+---
+
+# Gluten Support
+## Velox Backend
+
+[Gluten](https://github.com/oap-project/gluten) with velox backend supports Celeborn as remote shuffle service. Below introduction is used to enable this feature
+
+First refer to this URL(https://github.com/oap-project/gluten/blob/main/docs/get-started/Velox.md) to build Gluten with velox backend.
+
+When compiling the Gluten java module, you need to add rss, as follows:
+
+```
+mvn clean package -Pbackends-velox -Pspark-3.3 -Prss -DskipTests
+```
+
+Then add the paths of Gluten and Spark Celeborn Client packages to class path.

Review Comment:
   I think this part can be removed, the package name is not unique, there are different versions.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] zwangsheng commented on a diff in pull request #1784: [CELEBORN-870][DOC] Document on usage together with Gluten

Posted by "zwangsheng (via GitHub)" <gi...@apache.org>.
zwangsheng commented on code in PR #1784:
URL: https://github.com/apache/incubator-celeborn/pull/1784#discussion_r1281487866


##########
docs/developers/glutensupport.md:
##########
@@ -0,0 +1,62 @@
+---
+license: |
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+      https://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+---
+
+# Gluten Support
+## Velox Backend
+
+[Gluten](https://github.com/oap-project/gluten) with velox backend supports Celeborn as remote shuffle service. Below introduction is used to enable this feature
+
+First refer to this URL(https://github.com/oap-project/gluten/blob/main/docs/get-started/Velox.md) to build Gluten with velox backend.
+
+When compiling the Gluten Java module, it's required to enable `rss` profile, as follows:
+
+```
+mvn clean package -Pbackends-velox -Pspark-3.3 -Prss -DskipTests
+```
+
+Then add the Gluten and Spark Celeborn Client packages to your Spark application's classpath(usually add them into `$SPARK_HOME/jars`).
+
+- Celeborn: celeborn-client-spark-3-shaded_2.12-0.3.0-incubating.jar

Review Comment:
   There's a little doubt about that, we should use gluten compiled for `gluten-celeborn-XXX.jar` or `Celeborn-client-spark-3-shaded_2.12-XXX-incubating.jar`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] waitinfuture closed pull request #1784: [CELEBORN-870][DOC] Document on usage together with Gluten

Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture closed pull request #1784: [CELEBORN-870][DOC] Document on usage together with Gluten
URL: https://github.com/apache/incubator-celeborn/pull/1784


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] codecov[bot] commented on pull request #1784: [CELEBORN-870][DOC] Document on usage together with Gluten

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #1784:
URL: https://github.com/apache/incubator-celeborn/pull/1784#issuecomment-1661451373

   ## [Codecov](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1784?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   > Merging [#1784](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1784?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (d7d6a50) into [main](https://app.codecov.io/gh/apache/incubator-celeborn/commit/3e266c0cf6247245ab7bcd3e1abb49cf4ece0bca?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (3e266c0) will **not change** coverage.
   > Report is 1 commits behind head on main.
   > The diff coverage is `n/a`.
   
   ```diff
   @@           Coverage Diff           @@
   ##             main    #1784   +/-   ##
   =======================================
     Coverage   46.78%   46.78%           
   =======================================
     Files         162      162           
     Lines       10012    10012           
     Branches      923      923           
   =======================================
     Hits         4683     4683           
     Misses       5019     5019           
     Partials      310      310           
   ```
   
   
   [see 2 files with indirect coverage changes](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1784/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] kerwin-zk commented on a diff in pull request #1784: [CELEBORN-870][DOC] Document on usage together with Gluten

Posted by "kerwin-zk (via GitHub)" <gi...@apache.org>.
kerwin-zk commented on code in PR #1784:
URL: https://github.com/apache/incubator-celeborn/pull/1784#discussion_r1281505067


##########
docs/developers/glutensupport.md:
##########
@@ -0,0 +1,62 @@
+---
+license: |
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+      https://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+---
+
+# Gluten Support
+## Velox Backend
+
+[Gluten](https://github.com/oap-project/gluten) with velox backend supports Celeborn as remote shuffle service. Below introduction is used to enable this feature
+
+First refer to this URL(https://github.com/oap-project/gluten/blob/main/docs/get-started/Velox.md) to build Gluten with velox backend.
+
+When compiling the Gluten Java module, it's required to enable `rss` profile, as follows:
+
+```
+mvn clean package -Pbackends-velox -Pspark-3.3 -Prss -DskipTests
+```
+
+Then add the Gluten and Spark Celeborn Client packages to your Spark application's classpath(usually add them into `$SPARK_HOME/jars`).
+
+- Celeborn: celeborn-client-spark-3-shaded_2.12-0.3.0-incubating.jar

Review Comment:
   The previous Gluten+Celeborn integrated ```celeborn-client-spark-3-shaded_2.12-XXX-incubating.jar``` into the Gluten compilation package, so users only need the Gluten package before.  After this pr (https://github.com/oap-project/gluten/pull/2530) was merged, the Celeborn package was removed from Gluten, so users need to put ```celeborn-client-spark-3-shaded_2.12-XXX-incubating.jar```  in the class path or add it into `$SPARK_HOME/jars`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1784: [CELEBORN-870][DOC] Document on usage together with Gluten

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1784:
URL: https://github.com/apache/incubator-celeborn/pull/1784#discussion_r1281355531


##########
docs/developers/glutensupport.md:
##########
@@ -0,0 +1,51 @@
+---
+license: |
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+  http://www.apache.org/licenses/LICENSE-2.0

Review Comment:
   nit: fix the intend



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1784: [CELEBORN-870][DOC] Document on usage together with Gluten

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1784:
URL: https://github.com/apache/incubator-celeborn/pull/1784#discussion_r1281363127


##########
docs/developers/glutensupport.md:
##########
@@ -0,0 +1,51 @@
+---
+license: |
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+  http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+---
+
+# Gluten Support
+## Velox Backend
+
+[Gluten](https://github.com/oap-project/gluten) with velox backend supports Celeborn as remote shuffle service. Below introduction is used to enable this feature
+
+First refer to this URL(https://github.com/oap-project/gluten/blob/main/docs/get-started/Velox.md) to build Gluten with velox backend.
+
+When compiling the Gluten java module, you need to add rss, as follows:
+
+```
+mvn clean package -Pbackends-velox -Pspark-3.3 -Prss -DskipTests
+```
+
+Then add the paths of Gluten and Spark Celeborn Client packages to class path.

Review Comment:
   replace the `xxxx.jar` with real pattern ...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-celeborn] kerwin-zk commented on a diff in pull request #1784: [CELEBORN-870][DOC] Document on usage together with Gluten

Posted by "kerwin-zk (via GitHub)" <gi...@apache.org>.
kerwin-zk commented on code in PR #1784:
URL: https://github.com/apache/incubator-celeborn/pull/1784#discussion_r1281365316


##########
docs/developers/glutensupport.md:
##########
@@ -0,0 +1,51 @@
+---
+license: |
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+  http://www.apache.org/licenses/LICENSE-2.0
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+---
+
+# Gluten Support
+## Velox Backend
+
+[Gluten](https://github.com/oap-project/gluten) with velox backend supports Celeborn as remote shuffle service. Below introduction is used to enable this feature
+
+First refer to this URL(https://github.com/oap-project/gluten/blob/main/docs/get-started/Velox.md) to build Gluten with velox backend.
+
+When compiling the Gluten java module, you need to add rss, as follows:
+
+```
+mvn clean package -Pbackends-velox -Pspark-3.3 -Prss -DskipTests
+```
+
+Then add the paths of Gluten and Spark Celeborn Client packages to class path.

Review Comment:
   I think this part can be removed, the package name is not unique, there are different versions.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org