You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "PHILO-HE (via GitHub)" <gi...@apache.org> on 2025/01/07 06:03:13 UTC
[PR] [DOC] Update document [incubator-gluten]
PHILO-HE opened a new pull request, #8444:
URL: https://github.com/apache/incubator-gluten/pull/8444
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [DOC] Update document [incubator-gluten]
Posted by "srinivasst (via GitHub)" <gi...@apache.org>.
srinivasst commented on code in PR #8444:
URL: https://github.com/apache/incubator-gluten/pull/8444#discussion_r1906847760
##########
README.md:
##########
@@ -154,14 +141,9 @@ The below testing environment: a 8-nodes AWS cluster with 1TB data; Spark-3.1.1
Gluten is licensed under [Apache 2.0 license](https://www.apache.org/licenses/LICENSE-2.0).
-# 7 Contact
-
-Gluten was initiated by Intel and Kyligence in 2022. Several companies are also actively participating in the development, such as BIGO, Meituan, Alibaba Cloud, NetEase, Baidu, Microsoft, etc. If you are interested in Gluten project, please contact and subscribe below mailing lists for further discussion.
-
-* For community activity: dev@gluten.apache.org
-* For code repository activity: commits@gluten.apache.org
+# 7 Acknowledgements
-# 8 Thanks to our contributors
+Gluten was initiated by Intel and Kyligence in 2022. Several companies are also actively participating in the development, such as BIGO, Meituan, Alibaba Cloud, NetEase, Baidu, Microsoft, etc.
Review Comment:
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [DOC] Update document [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on code in PR #8444:
URL: https://github.com/apache/incubator-gluten/pull/8444#discussion_r1906020804
##########
README.md:
##########
@@ -63,58 +66,42 @@ There are several key components in Gluten:
# 3 How to Use
-There are two ways to use Gluten.
-
-# 3.1 Use Released Jar
-
-One way is to use released jar. Here is a simple example. Currently, only centos7/8 and ubuntu20.04/22.04 are well supported.
+Here is a basic configuration to enable Gluten in Spark.
```
-spark-shell \
- --master yarn --deploy-mode client \
- --conf spark.plugins=org.apache.gluten.GlutenPlugin \
- --conf spark.memory.offHeap.enabled=true \
- --conf spark.memory.offHeap.size=20g \
- --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
- --jars https://github.com/apache/incubator-gluten/releases/download/v1.1.1/gluten-velox-bundle-spark3.2_2.12-ubuntu_20.04_x86_64-1.1.1.jar
-```
-
-# 3.2 Custom Build
-
-Alternatively, you can build gluten from source, then do some configurations to enable Gluten plugin for Spark. Here is a simple example. Please refer to the corresponding backend part below for more details.
-
-```
-export gluten_jar = /PATH/TO/GLUTEN/backends-velox/target/<gluten-jar>
+export GLUTEN_JAR=/PATH/TO/GLUTEN_JAR
spark-shell \
--master yarn --deploy-mode client \
--conf spark.plugins=org.apache.gluten.GlutenPlugin \
--conf spark.memory.offHeap.enabled=true \
--conf spark.memory.offHeap.size=20g \
- --conf spark.driver.extraClassPath=${gluten_jar} \
- --conf spark.executor.extraClassPath=${gluten_jar} \
+ --conf spark.driver.extraClassPath=${GLUTEN_JAR} \
+ --conf spark.executor.extraClassPath=${GLUTEN_JAR} \
--conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager
...
```
-### 3.2.1 Build and install Gluten with Velox backend
+There are two ways to acquire Gluten jar.
+
+# 3.1 Use Released Jar (only applicable to Velox backend)
-If you want to use Gluten **Velox** backend, see [Build with Velox](./docs/get-started/Velox.md) to build and install the necessary libraries.
+Please download a tar package [here](https://downloads.apache.org/incubator/gluten/), then extract out Gluten jar from it.
Review Comment:
Is the jar build staticly? can the jar be used on any OS? I don't think so. If so let's put the OS where the jar is built.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [DOC] Update document [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE commented on code in PR #8444:
URL: https://github.com/apache/incubator-gluten/pull/8444#discussion_r1909732072
##########
README.md:
##########
@@ -63,58 +66,42 @@ There are several key components in Gluten:
# 3 How to Use
-There are two ways to use Gluten.
-
-# 3.1 Use Released Jar
-
-One way is to use released jar. Here is a simple example. Currently, only centos7/8 and ubuntu20.04/22.04 are well supported.
+Here is a basic configuration to enable Gluten in Spark.
```
-spark-shell \
- --master yarn --deploy-mode client \
- --conf spark.plugins=org.apache.gluten.GlutenPlugin \
- --conf spark.memory.offHeap.enabled=true \
- --conf spark.memory.offHeap.size=20g \
- --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
- --jars https://github.com/apache/incubator-gluten/releases/download/v1.1.1/gluten-velox-bundle-spark3.2_2.12-ubuntu_20.04_x86_64-1.1.1.jar
-```
-
-# 3.2 Custom Build
-
-Alternatively, you can build gluten from source, then do some configurations to enable Gluten plugin for Spark. Here is a simple example. Please refer to the corresponding backend part below for more details.
-
-```
-export gluten_jar = /PATH/TO/GLUTEN/backends-velox/target/<gluten-jar>
+export GLUTEN_JAR=/PATH/TO/GLUTEN_JAR
spark-shell \
--master yarn --deploy-mode client \
--conf spark.plugins=org.apache.gluten.GlutenPlugin \
--conf spark.memory.offHeap.enabled=true \
--conf spark.memory.offHeap.size=20g \
- --conf spark.driver.extraClassPath=${gluten_jar} \
- --conf spark.executor.extraClassPath=${gluten_jar} \
+ --conf spark.driver.extraClassPath=${GLUTEN_JAR} \
+ --conf spark.executor.extraClassPath=${GLUTEN_JAR} \
--conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager
...
```
-### 3.2.1 Build and install Gluten with Velox backend
+There are two ways to acquire Gluten jar.
+
+# 3.1 Use Released Jar (only applicable to Velox backend)
-If you want to use Gluten **Velox** backend, see [Build with Velox](./docs/get-started/Velox.md) to build and install the necessary libraries.
+Please download a tar package [here](https://downloads.apache.org/incubator/gluten/), then extract out Gluten jar from it.
Review Comment:
@FelixYBW, just confirmed with Weiting. The published release jar was generated by static build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [DOC] Update README.md [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE merged PR #8444:
URL: https://github.com/apache/incubator-gluten/pull/8444
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [DOC] Update document [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on PR #8444:
URL: https://github.com/apache/incubator-gluten/pull/8444#issuecomment-2578426264
@PHILO-HE can you add the slack channel as well? I just created one workspace for OSS Gluten:
[join.slack.com/t/osgluten/shared_invite/zt-2xn6px69t-SlO4eWnhw2YKFN87IsH7GQ](https://join.slack.com/t/osgluten/shared_invite/zt-2xn6px69t-SlO4eWnhw2YKFN87IsH7GQ)
The workspace:
[osgluten.slack.com](https://osgluten.slack.com/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [DOC] Update document [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on code in PR #8444:
URL: https://github.com/apache/incubator-gluten/pull/8444#discussion_r1906031485
##########
README.md:
##########
@@ -63,58 +66,42 @@ There are several key components in Gluten:
# 3 How to Use
-There are two ways to use Gluten.
-
-# 3.1 Use Released Jar
-
-One way is to use released jar. Here is a simple example. Currently, only centos7/8 and ubuntu20.04/22.04 are well supported.
+Here is a basic configuration to enable Gluten in Spark.
```
-spark-shell \
- --master yarn --deploy-mode client \
- --conf spark.plugins=org.apache.gluten.GlutenPlugin \
- --conf spark.memory.offHeap.enabled=true \
- --conf spark.memory.offHeap.size=20g \
- --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
- --jars https://github.com/apache/incubator-gluten/releases/download/v1.1.1/gluten-velox-bundle-spark3.2_2.12-ubuntu_20.04_x86_64-1.1.1.jar
-```
-
-# 3.2 Custom Build
-
-Alternatively, you can build gluten from source, then do some configurations to enable Gluten plugin for Spark. Here is a simple example. Please refer to the corresponding backend part below for more details.
-
-```
-export gluten_jar = /PATH/TO/GLUTEN/backends-velox/target/<gluten-jar>
+export GLUTEN_JAR=/PATH/TO/GLUTEN_JAR
spark-shell \
--master yarn --deploy-mode client \
--conf spark.plugins=org.apache.gluten.GlutenPlugin \
--conf spark.memory.offHeap.enabled=true \
--conf spark.memory.offHeap.size=20g \
- --conf spark.driver.extraClassPath=${gluten_jar} \
- --conf spark.executor.extraClassPath=${gluten_jar} \
+ --conf spark.driver.extraClassPath=${GLUTEN_JAR} \
+ --conf spark.executor.extraClassPath=${GLUTEN_JAR} \
--conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager
...
```
-### 3.2.1 Build and install Gluten with Velox backend
+There are two ways to acquire Gluten jar.
+
+# 3.1 Use Released Jar (only applicable to Velox backend)
-If you want to use Gluten **Velox** backend, see [Build with Velox](./docs/get-started/Velox.md) to build and install the necessary libraries.
+Please download a tar package [here](https://downloads.apache.org/incubator/gluten/), then extract out Gluten jar from it.
-### 3.2.2 Build and install Gluten with ClickHouse backend
+# 3.2 Custom Build
-If you want to use Gluten **ClickHouse** backend, see [Build with ClickHouse Backend](./docs/get-started/ClickHouse.md). ClickHouse backend is developed by [Kyligence](https://kyligence.io/), please visit https://github.com/Kyligence/ClickHouse for more infomation.
+For **Velox** backend, please refer to [Velox.md](./docs/get-started/Velox.md) and [build-guide.md](./docs/get-started/build-guide.md).
-### 3.2.3 Build options
+For **ClickHouse** backend, please refer to [ClickHouse.md](./docs/get-started/ClickHouse.md). ClickHouse backend is developed by [Kyligence](https://kyligence.io/), please visit https://github.com/Kyligence/ClickHouse for more information.
-See [Gluten build guide](./docs/get-started/build-guide.md).
+Gluten jar will be generated under `/PATH/TO/GLUTEN/package/target/` after build.
# 4 Contribution
-Welcome to contribute to Gluten project! See [contributing guide](CONTRIBUTING.md) about how to make contributions.
+Welcome to contribute to Gluten project! See [CONTRIBUTING.md](CONTRIBUTING.md) about how to make contributions.
## 4.1 Community
-Gluten successfully joined Apache Incubator since March'24. We welcome developers and users who are interested in Gluten project. Here are several ways to contact us:
+Gluten successfully became Apache incubator project in March'24. We welcome all developers and users who are interested in this project. Here are several ways to contact us:
Review Comment:
Add the community mail list here:
\* For community activity: dev@gluten.apache.org
\* For code repository activity: commits@gluten.apache.org
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [DOC] Update document [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #8444:
URL: https://github.com/apache/incubator-gluten/pull/8444#issuecomment-2579414674
> > > @PHILO-HE can you add the slack channel as well?
> > > ASF workspace invite link ([join.slack.com/t/the-asf/shared_invite/zt-2x74bfj6u-_3mH6Njlq6lZoIAZKNsutw](https://join.slack.com/t/the-asf/shared_invite/zt-2x74bfj6u-_3mH6Njlq6lZoIAZKNsutw)))
> > > Then search "incubator-gluten“ channel
> >
> >
> > For the slack channels (as well as other chat rooms), I recommend to lower the importance of them when users request for a way to ask questions or discuss about issues. Could encourage them to use GitHub or Apache mailing list more since they are more public and can be indexed by search engines. Do you guys have related thoughts?
> > cc @zhouyuan @weiting-chen
>
> They are complementary each other, not replace each other. Let's put the link on home page
I will be fine with adding a Slack link to documentation, since it's still open to all users so worth a try. Do you think we can add a note around the link to remind user that use of GitHub and Apache mailing list is encouraged?
E.g.,
```
* Slack channel: join.slack.com/...
Note: Slack channel should only be used for informal chats. It's encouraged to use GitHub / Mailing list for raising issues or technical discussions.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [DOC] Update README.md [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE commented on PR #8444:
URL: https://github.com/apache/incubator-gluten/pull/8444#issuecomment-2581891645
@FelixYBW, @zhztheplayer, if you have any other comment, please let me know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [DOC] Update document [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8444:
URL: https://github.com/apache/incubator-gluten/pull/8444#issuecomment-2574456657
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICEGJ-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Thanks for opening a pull request!
Could you open an issue for this pull request on Github Issues?
https://github.com/apache/incubator-gluten/issues
Then could you also rename ***commit message*** and ***pull request title*** in the following format?
[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}
See also:
* [Other pull requests](https://github.com/apache/incubator-gluten/pulls/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [DOC] Update document [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on code in PR #8444:
URL: https://github.com/apache/incubator-gluten/pull/8444#discussion_r1906030074
##########
README.md:
##########
@@ -154,14 +141,9 @@ The below testing environment: a 8-nodes AWS cluster with 1TB data; Spark-3.1.1
Gluten is licensed under [Apache 2.0 license](https://www.apache.org/licenses/LICENSE-2.0).
-# 7 Contact
-
-Gluten was initiated by Intel and Kyligence in 2022. Several companies are also actively participating in the development, such as BIGO, Meituan, Alibaba Cloud, NetEase, Baidu, Microsoft, etc. If you are interested in Gluten project, please contact and subscribe below mailing lists for further discussion.
-
-* For community activity: dev@gluten.apache.org
-* For code repository activity: commits@gluten.apache.org
+# 7 Acknowledgements
-# 8 Thanks to our contributors
+Gluten was initiated by Intel and Kyligence in 2022. Several companies are also actively participating in the development, such as BIGO, Meituan, Alibaba Cloud, NetEase, Baidu, Microsoft, etc.
Review Comment:
Add IBM here.
@srinivasst would like to add "Google" here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [DOC] Update document [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on code in PR #8444:
URL: https://github.com/apache/incubator-gluten/pull/8444#discussion_r1906020295
##########
README.md:
##########
@@ -63,58 +66,42 @@ There are several key components in Gluten:
# 3 How to Use
-There are two ways to use Gluten.
-
-# 3.1 Use Released Jar
-
-One way is to use released jar. Here is a simple example. Currently, only centos7/8 and ubuntu20.04/22.04 are well supported.
+Here is a basic configuration to enable Gluten in Spark.
```
-spark-shell \
- --master yarn --deploy-mode client \
- --conf spark.plugins=org.apache.gluten.GlutenPlugin \
- --conf spark.memory.offHeap.enabled=true \
- --conf spark.memory.offHeap.size=20g \
- --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
- --jars https://github.com/apache/incubator-gluten/releases/download/v1.1.1/gluten-velox-bundle-spark3.2_2.12-ubuntu_20.04_x86_64-1.1.1.jar
-```
-
-# 3.2 Custom Build
-
-Alternatively, you can build gluten from source, then do some configurations to enable Gluten plugin for Spark. Here is a simple example. Please refer to the corresponding backend part below for more details.
-
-```
-export gluten_jar = /PATH/TO/GLUTEN/backends-velox/target/<gluten-jar>
+export GLUTEN_JAR=/PATH/TO/GLUTEN_JAR
spark-shell \
--master yarn --deploy-mode client \
--conf spark.plugins=org.apache.gluten.GlutenPlugin \
--conf spark.memory.offHeap.enabled=true \
--conf spark.memory.offHeap.size=20g \
- --conf spark.driver.extraClassPath=${gluten_jar} \
- --conf spark.executor.extraClassPath=${gluten_jar} \
+ --conf spark.driver.extraClassPath=${GLUTEN_JAR} \
+ --conf spark.executor.extraClassPath=${GLUTEN_JAR} \
--conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager
...
```
-### 3.2.1 Build and install Gluten with Velox backend
+There are two ways to acquire Gluten jar.
+
+# 3.1 Use Released Jar (only applicable to Velox backend)
Review Comment:
"currently only applicable to Velox backend"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [DOC] Update README.md [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on code in PR #8444:
URL: https://github.com/apache/incubator-gluten/pull/8444#discussion_r1910584707
##########
README.md:
##########
@@ -63,58 +66,42 @@ There are several key components in Gluten:
# 3 How to Use
-There are two ways to use Gluten.
-
-# 3.1 Use Released Jar
-
-One way is to use released jar. Here is a simple example. Currently, only centos7/8 and ubuntu20.04/22.04 are well supported.
+Here is a basic configuration to enable Gluten in Spark.
```
-spark-shell \
- --master yarn --deploy-mode client \
- --conf spark.plugins=org.apache.gluten.GlutenPlugin \
- --conf spark.memory.offHeap.enabled=true \
- --conf spark.memory.offHeap.size=20g \
- --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
- --jars https://github.com/apache/incubator-gluten/releases/download/v1.1.1/gluten-velox-bundle-spark3.2_2.12-ubuntu_20.04_x86_64-1.1.1.jar
-```
-
-# 3.2 Custom Build
-
-Alternatively, you can build gluten from source, then do some configurations to enable Gluten plugin for Spark. Here is a simple example. Please refer to the corresponding backend part below for more details.
-
-```
-export gluten_jar = /PATH/TO/GLUTEN/backends-velox/target/<gluten-jar>
+export GLUTEN_JAR=/PATH/TO/GLUTEN_JAR
spark-shell \
--master yarn --deploy-mode client \
--conf spark.plugins=org.apache.gluten.GlutenPlugin \
--conf spark.memory.offHeap.enabled=true \
--conf spark.memory.offHeap.size=20g \
- --conf spark.driver.extraClassPath=${gluten_jar} \
- --conf spark.executor.extraClassPath=${gluten_jar} \
+ --conf spark.driver.extraClassPath=${GLUTEN_JAR} \
+ --conf spark.executor.extraClassPath=${GLUTEN_JAR} \
--conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager
...
```
-### 3.2.1 Build and install Gluten with Velox backend
+There are two ways to acquire Gluten jar.
+
+# 3.1 Use Released Jar (only applicable to Velox backend)
-If you want to use Gluten **Velox** backend, see [Build with Velox](./docs/get-started/Velox.md) to build and install the necessary libraries.
+Please download a tar package [here](https://downloads.apache.org/incubator/gluten/), then extract out Gluten jar from it.
Review Comment:
Even it's static built, it can still be used on the OS where it's built. Because it still depends on libc etc. which is different in different OS. We need to put the OSes we confirmed it can run on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [DOC] Update document [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #8444:
URL: https://github.com/apache/incubator-gluten/pull/8444#issuecomment-2579116609
> @PHILO-HE can you add the slack channel as well?
>
> ASF workspace invite link (https://join.slack.com/t/the-asf/shared_invite/zt-2x74bfj6u-_3mH6Njlq6lZoIAZKNsutw))
>
> Then search "incubator-gluten“ channel
For the slack channels (as well as other chat rooms), I recommend to lower the importance of them while users request for a way to ask questions or discuss about issues. Could encourage them to use GitHub or Apache mailing list more since they are more public and can be indexed by search engines. Do you guys have related thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [DOC] Update document [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on PR #8444:
URL: https://github.com/apache/incubator-gluten/pull/8444#issuecomment-2580824725
> I will be fine with adding a Slack link to documentation, since Slack's still open to all users so worth try with. Do you think we can add a note around the link to remind user that use of GitHub and Apache mailing list is encouraged?
We needn't. We should encourage user use github issues instead of mailing. Many companies use Slack more than mails now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [DOC] Update document [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on code in PR #8444:
URL: https://github.com/apache/incubator-gluten/pull/8444#discussion_r1906024290
##########
README.md:
##########
@@ -63,58 +66,42 @@ There are several key components in Gluten:
# 3 How to Use
-There are two ways to use Gluten.
-
Review Comment:
Add the supported Spark versions (including minor version) and OS. Highlight that with static build we support all the OS types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [DOC] Update document [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on code in PR #8444:
URL: https://github.com/apache/incubator-gluten/pull/8444#discussion_r1906023301
##########
README.md:
##########
@@ -63,58 +66,42 @@ There are several key components in Gluten:
# 3 How to Use
Review Comment:
Change it as "User Guide"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [DOC] Update document [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on code in PR #8444:
URL: https://github.com/apache/incubator-gluten/pull/8444#discussion_r1907587596
##########
README.md:
##########
@@ -63,58 +66,42 @@ There are several key components in Gluten:
# 3 How to Use
-There are two ways to use Gluten.
-
-# 3.1 Use Released Jar
-
-One way is to use released jar. Here is a simple example. Currently, only centos7/8 and ubuntu20.04/22.04 are well supported.
+Here is a basic configuration to enable Gluten in Spark.
```
-spark-shell \
- --master yarn --deploy-mode client \
- --conf spark.plugins=org.apache.gluten.GlutenPlugin \
- --conf spark.memory.offHeap.enabled=true \
- --conf spark.memory.offHeap.size=20g \
- --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
- --jars https://github.com/apache/incubator-gluten/releases/download/v1.1.1/gluten-velox-bundle-spark3.2_2.12-ubuntu_20.04_x86_64-1.1.1.jar
-```
-
-# 3.2 Custom Build
-
-Alternatively, you can build gluten from source, then do some configurations to enable Gluten plugin for Spark. Here is a simple example. Please refer to the corresponding backend part below for more details.
-
-```
-export gluten_jar = /PATH/TO/GLUTEN/backends-velox/target/<gluten-jar>
+export GLUTEN_JAR=/PATH/TO/GLUTEN_JAR
spark-shell \
--master yarn --deploy-mode client \
--conf spark.plugins=org.apache.gluten.GlutenPlugin \
--conf spark.memory.offHeap.enabled=true \
--conf spark.memory.offHeap.size=20g \
- --conf spark.driver.extraClassPath=${gluten_jar} \
- --conf spark.executor.extraClassPath=${gluten_jar} \
+ --conf spark.driver.extraClassPath=${GLUTEN_JAR} \
+ --conf spark.executor.extraClassPath=${GLUTEN_JAR} \
--conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager
...
```
-### 3.2.1 Build and install Gluten with Velox backend
+There are two ways to acquire Gluten jar.
+
+# 3.1 Use Released Jar (only applicable to Velox backend)
-If you want to use Gluten **Velox** backend, see [Build with Velox](./docs/get-started/Velox.md) to build and install the necessary libraries.
+Please download a tar package [here](https://downloads.apache.org/incubator/gluten/), then extract out Gluten jar from it.
-### 3.2.2 Build and install Gluten with ClickHouse backend
+# 3.2 Custom Build
-If you want to use Gluten **ClickHouse** backend, see [Build with ClickHouse Backend](./docs/get-started/ClickHouse.md). ClickHouse backend is developed by [Kyligence](https://kyligence.io/), please visit https://github.com/Kyligence/ClickHouse for more infomation.
+For **Velox** backend, please refer to [Velox.md](./docs/get-started/Velox.md) and [build-guide.md](./docs/get-started/build-guide.md).
-### 3.2.3 Build options
+For **ClickHouse** backend, please refer to [ClickHouse.md](./docs/get-started/ClickHouse.md). ClickHouse backend is developed by [Kyligence](https://kyligence.io/), please visit https://github.com/Kyligence/ClickHouse for more information.
-See [Gluten build guide](./docs/get-started/build-guide.md).
+Gluten jar will be generated under `/PATH/TO/GLUTEN/package/target/` after build.
# 4 Contribution
-Welcome to contribute to Gluten project! See [contributing guide](CONTRIBUTING.md) about how to make contributions.
+Welcome to contribute to Gluten project! See [CONTRIBUTING.md](CONTRIBUTING.md) about how to make contributions.
## 4.1 Community
-Gluten successfully joined Apache Incubator since March'24. We welcome developers and users who are interested in Gluten project. Here are several ways to contact us:
+Gluten successfully became Apache incubator project in March'24. We welcome all developers and users who are interested in this project. Here are several ways to contact us:
Review Comment:
Ignore it. Just noted the mailing lists section below
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [DOC] Update document [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on PR #8444:
URL: https://github.com/apache/incubator-gluten/pull/8444#issuecomment-2579338969
> > @PHILO-HE can you add the slack channel as well?
> > ASF workspace invite link ([join.slack.com/t/the-asf/shared\_invite/zt-2x74bfj6u-\_3mH6Njlq6lZoIAZKNsutw](https://join.slack.com/t/the-asf/shared_invite/zt-2x74bfj6u-_3mH6Njlq6lZoIAZKNsutw)))
> > Then search "incubator-gluten“ channel
>
> For the slack channels (as well as other chat rooms), I recommend to lower the importance of them when users request for a way to ask questions or discuss about issues. Could encourage them to use GitHub or Apache mailing list more since they are more public and can be indexed by search engines. Do you guys have related thoughts?
>
> cc @zhouyuan @weiting-chen
They are complementary each other, not replace each other. Let's put the link on home page
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [DOC] Update README.md [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on PR #8444:
URL: https://github.com/apache/incubator-gluten/pull/8444#issuecomment-2583088842
Looks good. @baibaichen @zzcclp Would you take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org