You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "kerwin-zk (via GitHub)" <gi...@apache.org> on 2023/08/02 03:45:43 UTC
[GitHub] [incubator-celeborn] kerwin-zk opened a new pull request, #1784: [CELEBORN-870][DOC] Document on usage together with Gluten
kerwin-zk opened a new pull request, #1784:
URL: https://github.com/apache/incubator-celeborn/pull/1784
### What changes were proposed in this pull request?
As title.
### Why are the changes needed?
As title.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manual test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1784: [CELEBORN-870][DOC] Document on usage together with Gluten
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1784:
URL: https://github.com/apache/incubator-celeborn/pull/1784#discussion_r1281357708
##########
docs/developers/glutensupport.md:
##########
@@ -0,0 +1,51 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Gluten Support
+## Velox Backend
+
+[Gluten](https://github.com/oap-project/gluten) with velox backend supports Celeborn as remote shuffle service. Below introduction is used to enable this feature
+
+First refer to this URL(https://github.com/oap-project/gluten/blob/main/docs/get-started/Velox.md) to build Gluten with velox backend.
+
+When compiling the Gluten java module, you need to add rss, as follows:
+
+```
+mvn clean package -Pbackends-velox -Pspark-3.3 -Prss -DskipTests
+```
+
+Then add the paths of Gluten and Spark Celeborn Client packages to class path.
+
+Currently to use Gluten following configurations are required in spark-defaults.conf
Review Comment:
```suggestion
Currently to use Gluten following configurations are required in `spark-defaults.conf`
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on pull request #1784: [CELEBORN-870][DOC] Document on usage together with Gluten
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #1784:
URL: https://github.com/apache/incubator-celeborn/pull/1784#issuecomment-1661451122
it's better to document the version matrix too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1784: [CELEBORN-870][DOC] Document on usage together with Gluten
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1784:
URL: https://github.com/apache/incubator-celeborn/pull/1784#discussion_r1281359164
##########
docs/developers/glutensupport.md:
##########
@@ -0,0 +1,51 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Gluten Support
+## Velox Backend
+
+[Gluten](https://github.com/oap-project/gluten) with velox backend supports Celeborn as remote shuffle service. Below introduction is used to enable this feature
+
+First refer to this URL(https://github.com/oap-project/gluten/blob/main/docs/get-started/Velox.md) to build Gluten with velox backend.
+
+When compiling the Gluten java module, you need to add rss, as follows:
+
+```
+mvn clean package -Pbackends-velox -Pspark-3.3 -Prss -DskipTests
+```
+
+Then add the paths of Gluten and Spark Celeborn Client packages to class path.
Review Comment:
```suggestion
Then add the Gluten and Spark Celeborn Client packages to your Spark application's classpath(usually add them into `$SPARK_HOME/jars`).
- Celeborn: xxxx.jar
- Gluten: xxx.jar
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1784: [CELEBORN-870][DOC] Document on usage together with Gluten
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1784:
URL: https://github.com/apache/incubator-celeborn/pull/1784#discussion_r1281356150
##########
docs/developers/glutensupport.md:
##########
@@ -0,0 +1,51 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Gluten Support
+## Velox Backend
+
+[Gluten](https://github.com/oap-project/gluten) with velox backend supports Celeborn as remote shuffle service. Below introduction is used to enable this feature
+
+First refer to this URL(https://github.com/oap-project/gluten/blob/main/docs/get-started/Velox.md) to build Gluten with velox backend.
+
+When compiling the Gluten java module, you need to add rss, as follows:
Review Comment:
```suggestion
When compiling the Gluten Java module, it's required to enable `rss` profile, as follows:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] kerwin-zk commented on a diff in pull request #1784: [CELEBORN-870][DOC] Document on usage together with Gluten
Posted by "kerwin-zk (via GitHub)" <gi...@apache.org>.
kerwin-zk commented on code in PR #1784:
URL: https://github.com/apache/incubator-celeborn/pull/1784#discussion_r1281365316
##########
docs/developers/glutensupport.md:
##########
@@ -0,0 +1,51 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Gluten Support
+## Velox Backend
+
+[Gluten](https://github.com/oap-project/gluten) with velox backend supports Celeborn as remote shuffle service. Below introduction is used to enable this feature
+
+First refer to this URL(https://github.com/oap-project/gluten/blob/main/docs/get-started/Velox.md) to build Gluten with velox backend.
+
+When compiling the Gluten java module, you need to add rss, as follows:
+
+```
+mvn clean package -Pbackends-velox -Pspark-3.3 -Prss -DskipTests
+```
+
+Then add the paths of Gluten and Spark Celeborn Client packages to class path.
Review Comment:
I think this part can be removed, the package name is not unique, there are different versions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] zwangsheng commented on a diff in pull request #1784: [CELEBORN-870][DOC] Document on usage together with Gluten
Posted by "zwangsheng (via GitHub)" <gi...@apache.org>.
zwangsheng commented on code in PR #1784:
URL: https://github.com/apache/incubator-celeborn/pull/1784#discussion_r1281487866
##########
docs/developers/glutensupport.md:
##########
@@ -0,0 +1,62 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Gluten Support
+## Velox Backend
+
+[Gluten](https://github.com/oap-project/gluten) with velox backend supports Celeborn as remote shuffle service. Below introduction is used to enable this feature
+
+First refer to this URL(https://github.com/oap-project/gluten/blob/main/docs/get-started/Velox.md) to build Gluten with velox backend.
+
+When compiling the Gluten Java module, it's required to enable `rss` profile, as follows:
+
+```
+mvn clean package -Pbackends-velox -Pspark-3.3 -Prss -DskipTests
+```
+
+Then add the Gluten and Spark Celeborn Client packages to your Spark application's classpath(usually add them into `$SPARK_HOME/jars`).
+
+- Celeborn: celeborn-client-spark-3-shaded_2.12-0.3.0-incubating.jar
Review Comment:
There's a little doubt about that, we should use gluten compiled for `gluten-celeborn-XXX.jar` or `Celeborn-client-spark-3-shaded_2.12-XXX-incubating.jar`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] waitinfuture closed pull request #1784: [CELEBORN-870][DOC] Document on usage together with Gluten
Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture closed pull request #1784: [CELEBORN-870][DOC] Document on usage together with Gluten
URL: https://github.com/apache/incubator-celeborn/pull/1784
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] codecov[bot] commented on pull request #1784: [CELEBORN-870][DOC] Document on usage together with Gluten
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #1784:
URL: https://github.com/apache/incubator-celeborn/pull/1784#issuecomment-1661451373
## [Codecov](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1784?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#1784](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1784?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (d7d6a50) into [main](https://app.codecov.io/gh/apache/incubator-celeborn/commit/3e266c0cf6247245ab7bcd3e1abb49cf4ece0bca?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (3e266c0) will **not change** coverage.
> Report is 1 commits behind head on main.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## main #1784 +/- ##
=======================================
Coverage 46.78% 46.78%
=======================================
Files 162 162
Lines 10012 10012
Branches 923 923
=======================================
Hits 4683 4683
Misses 5019 5019
Partials 310 310
```
[see 2 files with indirect coverage changes](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1784/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] kerwin-zk commented on a diff in pull request #1784: [CELEBORN-870][DOC] Document on usage together with Gluten
Posted by "kerwin-zk (via GitHub)" <gi...@apache.org>.
kerwin-zk commented on code in PR #1784:
URL: https://github.com/apache/incubator-celeborn/pull/1784#discussion_r1281505067
##########
docs/developers/glutensupport.md:
##########
@@ -0,0 +1,62 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Gluten Support
+## Velox Backend
+
+[Gluten](https://github.com/oap-project/gluten) with velox backend supports Celeborn as remote shuffle service. Below introduction is used to enable this feature
+
+First refer to this URL(https://github.com/oap-project/gluten/blob/main/docs/get-started/Velox.md) to build Gluten with velox backend.
+
+When compiling the Gluten Java module, it's required to enable `rss` profile, as follows:
+
+```
+mvn clean package -Pbackends-velox -Pspark-3.3 -Prss -DskipTests
+```
+
+Then add the Gluten and Spark Celeborn Client packages to your Spark application's classpath(usually add them into `$SPARK_HOME/jars`).
+
+- Celeborn: celeborn-client-spark-3-shaded_2.12-0.3.0-incubating.jar
Review Comment:
The previous Gluten+Celeborn integrated ```celeborn-client-spark-3-shaded_2.12-XXX-incubating.jar``` into the Gluten compilation package, so users only need the Gluten package before. After this pr (https://github.com/oap-project/gluten/pull/2530) was merged, the Celeborn package was removed from Gluten, so users need to put ```celeborn-client-spark-3-shaded_2.12-XXX-incubating.jar``` in the class path or add it into `$SPARK_HOME/jars`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1784: [CELEBORN-870][DOC] Document on usage together with Gluten
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1784:
URL: https://github.com/apache/incubator-celeborn/pull/1784#discussion_r1281355531
##########
docs/developers/glutensupport.md:
##########
@@ -0,0 +1,51 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
Review Comment:
nit: fix the intend
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] pan3793 commented on a diff in pull request #1784: [CELEBORN-870][DOC] Document on usage together with Gluten
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #1784:
URL: https://github.com/apache/incubator-celeborn/pull/1784#discussion_r1281363127
##########
docs/developers/glutensupport.md:
##########
@@ -0,0 +1,51 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Gluten Support
+## Velox Backend
+
+[Gluten](https://github.com/oap-project/gluten) with velox backend supports Celeborn as remote shuffle service. Below introduction is used to enable this feature
+
+First refer to this URL(https://github.com/oap-project/gluten/blob/main/docs/get-started/Velox.md) to build Gluten with velox backend.
+
+When compiling the Gluten java module, you need to add rss, as follows:
+
+```
+mvn clean package -Pbackends-velox -Pspark-3.3 -Prss -DskipTests
+```
+
+Then add the paths of Gluten and Spark Celeborn Client packages to class path.
Review Comment:
replace the `xxxx.jar` with real pattern ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] kerwin-zk commented on a diff in pull request #1784: [CELEBORN-870][DOC] Document on usage together with Gluten
Posted by "kerwin-zk (via GitHub)" <gi...@apache.org>.
kerwin-zk commented on code in PR #1784:
URL: https://github.com/apache/incubator-celeborn/pull/1784#discussion_r1281365316
##########
docs/developers/glutensupport.md:
##########
@@ -0,0 +1,51 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Gluten Support
+## Velox Backend
+
+[Gluten](https://github.com/oap-project/gluten) with velox backend supports Celeborn as remote shuffle service. Below introduction is used to enable this feature
+
+First refer to this URL(https://github.com/oap-project/gluten/blob/main/docs/get-started/Velox.md) to build Gluten with velox backend.
+
+When compiling the Gluten java module, you need to add rss, as follows:
+
+```
+mvn clean package -Pbackends-velox -Pspark-3.3 -Prss -DskipTests
+```
+
+Then add the paths of Gluten and Spark Celeborn Client packages to class path.
Review Comment:
I think this part can be removed, the package name is not unique, there are different versions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org