You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "ulysses-you (via GitHub)" <gi...@apache.org> on 2024/03/26 03:33:18 UTC
[PR] [CORE] Support JDK17 [incubator-gluten]
ulysses-you opened a new pull request, #5120:
URL: https://github.com/apache/incubator-gluten/pull/5120
## What changes were proposed in this pull request?
- Add jdk-17 maven profile
- Add jdk17 tpcds/tpch github action workflow
- Update docs
## How was this patch tested?
Pass CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Support JDK17 [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5120:
URL: https://github.com/apache/incubator-gluten/pull/5120#issuecomment-2019918106
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Support JDK17 [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5120:
URL: https://github.com/apache/incubator-gluten/pull/5120#issuecomment-2021727351
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Support JDK17 [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5120:
URL: https://github.com/apache/incubator-gluten/pull/5120#issuecomment-2019369826
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Support JDK17 [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5120:
URL: https://github.com/apache/incubator-gluten/pull/5120#issuecomment-2019321092
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICEGJ-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Thanks for opening a pull request!
Could you open an issue for this pull request on Github Issues?
https://github.com/apache/incubator-gluten/issues
Then could you also rename ***commit message*** and ***pull request title*** in the following format?
[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}
See also:
* [Other pull requests](https://github.com/apache/incubator-gluten/pulls/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Support JDK17 [incubator-gluten]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on code in PR #5120:
URL: https://github.com/apache/incubator-gluten/pull/5120#discussion_r1538964666
##########
.github/workflows/velox_docker.yml:
##########
@@ -155,32 +146,39 @@ jobs:
name: velox-native-lib-${{github.sha}}
path: ./cpp/build/releases
- name: Update mirror list
+ if: matrix.os == 'centos:8'
Review Comment:
I combine `run-tpc-test-centos7` and `run-tpc-test-centos8`. It seems the main difference is mirror.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Support JDK17 [incubator-gluten]
Posted by "WangGuangxin (via GitHub)" <gi...@apache.org>.
WangGuangxin commented on PR #5120:
URL: https://github.com/apache/incubator-gluten/pull/5120#issuecomment-2173139018
@ulysses-you Do you encountered following error when switch from JDK8 to JDK17
```shell
free(): invalid pointer
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Support JDK17 [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5120:
URL: https://github.com/apache/incubator-gluten/pull/5120#issuecomment-2019472931
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Support JDK17 [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5120:
URL: https://github.com/apache/incubator-gluten/pull/5120#issuecomment-2019477532
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Support JDK17 [incubator-gluten]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn merged PR #5120:
URL: https://github.com/apache/incubator-gluten/pull/5120
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Support JDK17 [incubator-gluten]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on code in PR #5120:
URL: https://github.com/apache/incubator-gluten/pull/5120#discussion_r1538574833
##########
.github/workflows/velox_docker.yml:
##########
@@ -84,31 +89,41 @@ jobs:
path: ./cpp/build/releases
- name: Setup java and maven
run: |
- apt-get update && \
- apt-get install -y openjdk-8-jdk maven && \
+ if [[ "${{ matrix.java }}" == "17" ]]; then
+ apt-get update && \
+ apt-get install -y openjdk-17-jdk maven
+ else
+ apt-get update && \
+ apt-get install -y openjdk-8-jdk maven
+ fi
apt remove openjdk-11* -y
- name: Build for Spark ${{ matrix.spark }}
run: |
cd $GITHUB_WORKSPACE/ && \
- mvn clean install -P${{ matrix.spark }} -Pbackends-velox -DskipTests
+ mvn clean install -P${{ matrix.spark }} -P${{ matrix.java }} -Pbackends-velox -DskipTests
- name: Build and run TPCH/DS ${{ matrix.spark }}
run: |
cd $GITHUB_WORKSPACE/tools/gluten-it && \
- mvn clean install -P${{ matrix.spark }} \
+ mvn clean install -P${{ matrix.spark }} -P${{ matrix.java }} \
&& GLUTEN_IT_JVM_ARGS=-Xmx5G sbin/gluten-it.sh queries-compare \
--local --preset=velox --benchmark-type=h --error-on-memleak --off-heap-size=10g -s=1.0 --threads=16 --iterations=1 \
&& GLUTEN_IT_JVM_ARGS=-Xmx5G sbin/gluten-it.sh queries-compare \
--local --preset=velox --benchmark-type=ds --error-on-memleak --off-heap-size=10g -s=1.0 --threads=16 --iterations=1
-
- run-tpc-test-centos7:
+ run-tpc-test-centos:
needs: build-native-lib
strategy:
fail-fast: false
matrix:
+ os: ["centos:7", "centos:8"]
spark: ["spark-3.2", "spark-3.3", "spark-3.4", "spark-3.5"]
+ java: ["jdk-8", "jdk-17"]
Review Comment:
I'm not sure about it.. But seems it can work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Support JDK17 [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5120:
URL: https://github.com/apache/incubator-gluten/pull/5120#issuecomment-2019564540
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Support JDK17 [incubator-gluten]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on PR #5120:
URL: https://github.com/apache/incubator-gluten/pull/5120#issuecomment-2020149985
cc @zhztheplayer @zhouyuan @PHILO-HE if you have other comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Support JDK17 [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5120:
URL: https://github.com/apache/incubator-gluten/pull/5120#issuecomment-2021843842
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Support JDK17 [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5120:
URL: https://github.com/apache/incubator-gluten/pull/5120#discussion_r1538580041
##########
.github/workflows/velox_docker.yml:
##########
@@ -84,31 +89,41 @@ jobs:
path: ./cpp/build/releases
- name: Setup java and maven
run: |
- apt-get update && \
- apt-get install -y openjdk-8-jdk maven && \
+ if [[ "${{ matrix.java }}" == "17" ]]; then
+ apt-get update && \
+ apt-get install -y openjdk-17-jdk maven
+ else
+ apt-get update && \
+ apt-get install -y openjdk-8-jdk maven
+ fi
apt remove openjdk-11* -y
- name: Build for Spark ${{ matrix.spark }}
run: |
cd $GITHUB_WORKSPACE/ && \
- mvn clean install -P${{ matrix.spark }} -Pbackends-velox -DskipTests
+ mvn clean install -P${{ matrix.spark }} -P${{ matrix.java }} -Pbackends-velox -DskipTests
- name: Build and run TPCH/DS ${{ matrix.spark }}
run: |
cd $GITHUB_WORKSPACE/tools/gluten-it && \
- mvn clean install -P${{ matrix.spark }} \
+ mvn clean install -P${{ matrix.spark }} -P${{ matrix.java }} \
&& GLUTEN_IT_JVM_ARGS=-Xmx5G sbin/gluten-it.sh queries-compare \
--local --preset=velox --benchmark-type=h --error-on-memleak --off-heap-size=10g -s=1.0 --threads=16 --iterations=1 \
&& GLUTEN_IT_JVM_ARGS=-Xmx5G sbin/gluten-it.sh queries-compare \
--local --preset=velox --benchmark-type=ds --error-on-memleak --off-heap-size=10g -s=1.0 --threads=16 --iterations=1
-
- run-tpc-test-centos7:
+ run-tpc-test-centos:
needs: build-native-lib
strategy:
fail-fast: false
matrix:
+ os: ["centos:7", "centos:8"]
spark: ["spark-3.2", "spark-3.3", "spark-3.4", "spark-3.5"]
+ java: ["jdk-8", "jdk-17"]
Review Comment:
https://infra.apache.org/github-actions-policy.html
I wonder we may need to follow this policy from Apache infra. But not sure about it. Though it suggests concurrency <= 20.
Do we really need for both centos 7 / 8 in this job? Or is it still enough if we set a individual job for JDK 17 with out matrix?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Support JDK17 [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5120:
URL: https://github.com/apache/incubator-gluten/pull/5120#discussion_r1538603244
##########
.github/workflows/velox_docker.yml:
##########
@@ -84,31 +89,41 @@ jobs:
path: ./cpp/build/releases
- name: Setup java and maven
run: |
- apt-get update && \
- apt-get install -y openjdk-8-jdk maven && \
+ if [[ "${{ matrix.java }}" == "17" ]]; then
+ apt-get update && \
+ apt-get install -y openjdk-17-jdk maven
+ else
+ apt-get update && \
+ apt-get install -y openjdk-8-jdk maven
+ fi
apt remove openjdk-11* -y
- name: Build for Spark ${{ matrix.spark }}
run: |
cd $GITHUB_WORKSPACE/ && \
- mvn clean install -P${{ matrix.spark }} -Pbackends-velox -DskipTests
+ mvn clean install -P${{ matrix.spark }} -P${{ matrix.java }} -Pbackends-velox -DskipTests
- name: Build and run TPCH/DS ${{ matrix.spark }}
run: |
cd $GITHUB_WORKSPACE/tools/gluten-it && \
- mvn clean install -P${{ matrix.spark }} \
+ mvn clean install -P${{ matrix.spark }} -P${{ matrix.java }} \
&& GLUTEN_IT_JVM_ARGS=-Xmx5G sbin/gluten-it.sh queries-compare \
--local --preset=velox --benchmark-type=h --error-on-memleak --off-heap-size=10g -s=1.0 --threads=16 --iterations=1 \
&& GLUTEN_IT_JVM_ARGS=-Xmx5G sbin/gluten-it.sh queries-compare \
--local --preset=velox --benchmark-type=ds --error-on-memleak --off-heap-size=10g -s=1.0 --threads=16 --iterations=1
-
- run-tpc-test-centos7:
+ run-tpc-test-centos:
needs: build-native-lib
strategy:
fail-fast: false
matrix:
+ os: ["centos:7", "centos:8"]
spark: ["spark-3.2", "spark-3.3", "spark-3.4", "spark-3.5"]
+ java: ["jdk-8", "jdk-17"]
Review Comment:
> yeah, we can reduce some matrix. How about only test centos8 + ubuntu20.04 ?
Sounds good if it's enough. Thanks.
> Is it a good time to drop supporting Spark3.2 ?
It was on the plan to stop supporting 3.2 but looks like it may require for some time or some discussions to take the action. cc @zhouyuan @weiting-chen may have more contexts on that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Support JDK17 [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #5120:
URL: https://github.com/apache/incubator-gluten/pull/5120#discussion_r1538572510
##########
.github/workflows/velox_docker.yml:
##########
@@ -84,31 +89,41 @@ jobs:
path: ./cpp/build/releases
- name: Setup java and maven
run: |
- apt-get update && \
- apt-get install -y openjdk-8-jdk maven && \
+ if [[ "${{ matrix.java }}" == "17" ]]; then
+ apt-get update && \
+ apt-get install -y openjdk-17-jdk maven
+ else
+ apt-get update && \
+ apt-get install -y openjdk-8-jdk maven
+ fi
apt remove openjdk-11* -y
- name: Build for Spark ${{ matrix.spark }}
run: |
cd $GITHUB_WORKSPACE/ && \
- mvn clean install -P${{ matrix.spark }} -Pbackends-velox -DskipTests
+ mvn clean install -P${{ matrix.spark }} -P${{ matrix.java }} -Pbackends-velox -DskipTests
- name: Build and run TPCH/DS ${{ matrix.spark }}
run: |
cd $GITHUB_WORKSPACE/tools/gluten-it && \
- mvn clean install -P${{ matrix.spark }} \
+ mvn clean install -P${{ matrix.spark }} -P${{ matrix.java }} \
&& GLUTEN_IT_JVM_ARGS=-Xmx5G sbin/gluten-it.sh queries-compare \
--local --preset=velox --benchmark-type=h --error-on-memleak --off-heap-size=10g -s=1.0 --threads=16 --iterations=1 \
&& GLUTEN_IT_JVM_ARGS=-Xmx5G sbin/gluten-it.sh queries-compare \
--local --preset=velox --benchmark-type=ds --error-on-memleak --off-heap-size=10g -s=1.0 --threads=16 --iterations=1
-
- run-tpc-test-centos7:
+ run-tpc-test-centos:
needs: build-native-lib
strategy:
fail-fast: false
matrix:
+ os: ["centos:7", "centos:8"]
spark: ["spark-3.2", "spark-3.3", "spark-3.4", "spark-3.5"]
+ java: ["jdk-8", "jdk-17"]
Review Comment:
Shouldn't GitHub have some limits on matrix config? It's like 2 * 4 * 2 = 16 jobs generated...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Support JDK17 [incubator-gluten]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on code in PR #5120:
URL: https://github.com/apache/incubator-gluten/pull/5120#discussion_r1540297481
##########
docs/get-started/Velox.md:
##########
@@ -5,28 +5,34 @@ nav_order: 1
parent: Getting-Started
---
# Supported Version
-| Type | Version |
-|-------|------------------------------|
-| Spark | 3.2.2, 3.3.1 |
-| OS | Ubuntu20.04/22.04, Centos7/8 |
-| jdk | openjdk8 |
-| scala | 2.12
-Spark3.4.0 support is still WIP. TPCH/DS can pass, UT is not yet passed.
+| Type | Version |
+|-------|---------------------------------|
+| Spark | 3.2.2, 3.3.1, 3.4.2, 3.5.1(wip) |
+| OS | Ubuntu20.04/22.04, Centos7/8 |
+| jdk | openjdk8/jdk17 |
+| scala | 2.12 |
-There are pending PRs for jdk11 support.
+**JDK17**
Review Comment:
I moved it to `NewToGluten.md`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Support JDK17 [incubator-gluten]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on code in PR #5120:
URL: https://github.com/apache/incubator-gluten/pull/5120#discussion_r1538596991
##########
.github/workflows/velox_docker.yml:
##########
@@ -84,31 +89,41 @@ jobs:
path: ./cpp/build/releases
- name: Setup java and maven
run: |
- apt-get update && \
- apt-get install -y openjdk-8-jdk maven && \
+ if [[ "${{ matrix.java }}" == "17" ]]; then
+ apt-get update && \
+ apt-get install -y openjdk-17-jdk maven
+ else
+ apt-get update && \
+ apt-get install -y openjdk-8-jdk maven
+ fi
apt remove openjdk-11* -y
- name: Build for Spark ${{ matrix.spark }}
run: |
cd $GITHUB_WORKSPACE/ && \
- mvn clean install -P${{ matrix.spark }} -Pbackends-velox -DskipTests
+ mvn clean install -P${{ matrix.spark }} -P${{ matrix.java }} -Pbackends-velox -DskipTests
- name: Build and run TPCH/DS ${{ matrix.spark }}
run: |
cd $GITHUB_WORKSPACE/tools/gluten-it && \
- mvn clean install -P${{ matrix.spark }} \
+ mvn clean install -P${{ matrix.spark }} -P${{ matrix.java }} \
&& GLUTEN_IT_JVM_ARGS=-Xmx5G sbin/gluten-it.sh queries-compare \
--local --preset=velox --benchmark-type=h --error-on-memleak --off-heap-size=10g -s=1.0 --threads=16 --iterations=1 \
&& GLUTEN_IT_JVM_ARGS=-Xmx5G sbin/gluten-it.sh queries-compare \
--local --preset=velox --benchmark-type=ds --error-on-memleak --off-heap-size=10g -s=1.0 --threads=16 --iterations=1
-
- run-tpc-test-centos7:
+ run-tpc-test-centos:
needs: build-native-lib
strategy:
fail-fast: false
matrix:
+ os: ["centos:7", "centos:8"]
spark: ["spark-3.2", "spark-3.3", "spark-3.4", "spark-3.5"]
+ java: ["jdk-8", "jdk-17"]
Review Comment:
yeah, we can reduce some matrix. How about only test centos8 + ubuntu20.04 ?
Is it a good time to drop supporting Spark3.2 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Support JDK17 [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE commented on code in PR #5120:
URL: https://github.com/apache/incubator-gluten/pull/5120#discussion_r1539216963
##########
.github/workflows/velox_docker.yml:
##########
@@ -73,6 +73,17 @@ jobs:
matrix:
os: ["ubuntu:20.04", "ubuntu:22.04"]
spark: ["spark-3.2", "spark-3.3", "spark-3.4", "spark-3.5"]
+ java: [ "java-8", "java-17" ]
+ # Spark supports JDK17 since 3.3 and later, see https://issues.apache.org/jira/browse/SPARK-33772
+ exclude:
Review Comment:
Looks `include` cannot make it concise. Please ignore this comment. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Support JDK17 [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5120:
URL: https://github.com/apache/incubator-gluten/pull/5120#issuecomment-2019321364
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Support JDK17 [incubator-gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on code in PR #5120:
URL: https://github.com/apache/incubator-gluten/pull/5120#discussion_r1538805865
##########
.github/workflows/velox_docker.yml:
##########
@@ -84,31 +89,41 @@ jobs:
path: ./cpp/build/releases
- name: Setup java and maven
run: |
- apt-get update && \
- apt-get install -y openjdk-8-jdk maven && \
+ if [[ "${{ matrix.java }}" == "17" ]]; then
+ apt-get update && \
+ apt-get install -y openjdk-17-jdk maven
+ else
+ apt-get update && \
+ apt-get install -y openjdk-8-jdk maven
+ fi
apt remove openjdk-11* -y
- name: Build for Spark ${{ matrix.spark }}
run: |
cd $GITHUB_WORKSPACE/ && \
- mvn clean install -P${{ matrix.spark }} -Pbackends-velox -DskipTests
+ mvn clean install -P${{ matrix.spark }} -P${{ matrix.java }} -Pbackends-velox -DskipTests
- name: Build and run TPCH/DS ${{ matrix.spark }}
run: |
cd $GITHUB_WORKSPACE/tools/gluten-it && \
- mvn clean install -P${{ matrix.spark }} \
+ mvn clean install -P${{ matrix.spark }} -P${{ matrix.java }} \
&& GLUTEN_IT_JVM_ARGS=-Xmx5G sbin/gluten-it.sh queries-compare \
--local --preset=velox --benchmark-type=h --error-on-memleak --off-heap-size=10g -s=1.0 --threads=16 --iterations=1 \
&& GLUTEN_IT_JVM_ARGS=-Xmx5G sbin/gluten-it.sh queries-compare \
--local --preset=velox --benchmark-type=ds --error-on-memleak --off-heap-size=10g -s=1.0 --threads=16 --iterations=1
-
- run-tpc-test-centos7:
+ run-tpc-test-centos:
needs: build-native-lib
strategy:
fail-fast: false
matrix:
+ os: ["centos:7", "centos:8"]
spark: ["spark-3.2", "spark-3.3", "spark-3.4", "spark-3.5"]
+ java: ["jdk-8", "jdk-17"]
Review Comment:
as far as I know spark3.2 is still active in some production env, so better to keep it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Support JDK17 [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE commented on code in PR #5120:
URL: https://github.com/apache/incubator-gluten/pull/5120#discussion_r1539159436
##########
.github/workflows/velox_docker.yml:
##########
@@ -84,69 +95,49 @@ jobs:
path: ./cpp/build/releases
- name: Setup java and maven
run: |
- apt-get update && \
- apt-get install -y openjdk-8-jdk maven && \
+ if [ "${{ matrix.java }}" = "java-17" ]; then
+ apt-get update && apt-get install -y openjdk-17-jdk maven
+ else
+ apt-get update && apt-get install -y openjdk-8-jdk maven
+ fi
apt remove openjdk-11* -y
- - name: Build for Spark ${{ matrix.spark }}
- run: |
- cd $GITHUB_WORKSPACE/ && \
- mvn clean install -P${{ matrix.spark }} -Pbackends-velox -DskipTests
- - name: Build and run TPCH/DS ${{ matrix.spark }}
- run: |
- cd $GITHUB_WORKSPACE/tools/gluten-it && \
- mvn clean install -P${{ matrix.spark }} \
- && GLUTEN_IT_JVM_ARGS=-Xmx5G sbin/gluten-it.sh queries-compare \
- --local --preset=velox --benchmark-type=h --error-on-memleak --off-heap-size=10g -s=1.0 --threads=16 --iterations=1 \
- && GLUTEN_IT_JVM_ARGS=-Xmx5G sbin/gluten-it.sh queries-compare \
- --local --preset=velox --benchmark-type=ds --error-on-memleak --off-heap-size=10g -s=1.0 --threads=16 --iterations=1
-
-
- run-tpc-test-centos7:
- needs: build-native-lib
- strategy:
- fail-fast: false
- matrix:
- spark: ["spark-3.2", "spark-3.3", "spark-3.4", "spark-3.5"]
- runs-on: ubuntu-20.04
- container: centos:7
- steps:
- - uses: actions/checkout@v2
- - name: Download All Artifacts
- uses: actions/download-artifact@v2
- with:
- name: velox-native-lib-${{github.sha}}
- path: ./cpp/build/releases
- - name: Setup java and maven
- run: |
- yum update -y && yum install -y java-1.8.0-openjdk-devel wget
- wget https://downloads.apache.org/maven/maven-3/3.8.8/binaries/apache-maven-3.8.8-bin.tar.gz
- tar -xvf apache-maven-3.8.8-bin.tar.gz
- mv apache-maven-3.8.8 /usr/lib/maven
- - name: Build for Spark ${{ matrix.spark }}
+ - name: Build and run TPCH/DS
run: |
cd $GITHUB_WORKSPACE/
- export MAVEN_HOME=/usr/lib/maven
- export PATH=${PATH}:${MAVEN_HOME}/bin
- mvn clean install -P${{ matrix.spark }} -Pbackends-velox -DskipTests
- - name: Build and run TPCH/DS ${{ matrix.spark }}
- run: |
- cd $GITHUB_WORKSPACE/tools/gluten-it
- export MAVEN_HOME=/usr/lib/maven
- export PATH=${PATH}:${MAVEN_HOME}/bin
- mvn clean install -P${{ matrix.spark }} \
+ if [ "${{ matrix.java }}" = "java-17" ]; then
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
Review Comment:
Nit:
export JAVA_HOME=/usr/lib/jvm/${{ matrix.java }}-openjdk-amd64
##########
.github/workflows/velox_docker.yml:
##########
@@ -73,6 +73,17 @@ jobs:
matrix:
os: ["ubuntu:20.04", "ubuntu:22.04"]
spark: ["spark-3.2", "spark-3.3", "spark-3.4", "spark-3.5"]
+ java: [ "java-8", "java-17" ]
+ # Spark supports JDK17 since 3.3 and later, see https://issues.apache.org/jira/browse/SPARK-33772
+ exclude:
Review Comment:
Nit: maybe, better to use `include` for simplicity.
##########
docs/get-started/Velox.md:
##########
@@ -5,28 +5,34 @@ nav_order: 1
parent: Getting-Started
---
# Supported Version
-| Type | Version |
-|-------|------------------------------|
-| Spark | 3.2.2, 3.3.1 |
-| OS | Ubuntu20.04/22.04, Centos7/8 |
-| jdk | openjdk8 |
-| scala | 2.12
-Spark3.4.0 support is still WIP. TPCH/DS can pass, UT is not yet passed.
+| Type | Version |
+|-------|---------------------------------|
+| Spark | 3.2.2, 3.3.1, 3.4.2, 3.5.1(wip) |
+| OS | Ubuntu20.04/22.04, Centos7/8 |
+| jdk | openjdk8/jdk17 |
+| scala | 2.12 |
-There are pending PRs for jdk11 support.
+**JDK17**
Review Comment:
Maybe, better to document this part in a common place, as it is not specific to Velox backend.
##########
.github/workflows/velox_docker.yml:
##########
@@ -84,69 +95,49 @@ jobs:
path: ./cpp/build/releases
- name: Setup java and maven
run: |
- apt-get update && \
- apt-get install -y openjdk-8-jdk maven && \
+ if [ "${{ matrix.java }}" = "java-17" ]; then
+ apt-get update && apt-get install -y openjdk-17-jdk maven
+ else
+ apt-get update && apt-get install -y openjdk-8-jdk maven
+ fi
apt remove openjdk-11* -y
- - name: Build for Spark ${{ matrix.spark }}
- run: |
- cd $GITHUB_WORKSPACE/ && \
- mvn clean install -P${{ matrix.spark }} -Pbackends-velox -DskipTests
- - name: Build and run TPCH/DS ${{ matrix.spark }}
- run: |
- cd $GITHUB_WORKSPACE/tools/gluten-it && \
- mvn clean install -P${{ matrix.spark }} \
- && GLUTEN_IT_JVM_ARGS=-Xmx5G sbin/gluten-it.sh queries-compare \
- --local --preset=velox --benchmark-type=h --error-on-memleak --off-heap-size=10g -s=1.0 --threads=16 --iterations=1 \
- && GLUTEN_IT_JVM_ARGS=-Xmx5G sbin/gluten-it.sh queries-compare \
- --local --preset=velox --benchmark-type=ds --error-on-memleak --off-heap-size=10g -s=1.0 --threads=16 --iterations=1
-
-
- run-tpc-test-centos7:
- needs: build-native-lib
- strategy:
- fail-fast: false
- matrix:
- spark: ["spark-3.2", "spark-3.3", "spark-3.4", "spark-3.5"]
- runs-on: ubuntu-20.04
- container: centos:7
- steps:
- - uses: actions/checkout@v2
- - name: Download All Artifacts
- uses: actions/download-artifact@v2
- with:
- name: velox-native-lib-${{github.sha}}
- path: ./cpp/build/releases
- - name: Setup java and maven
- run: |
- yum update -y && yum install -y java-1.8.0-openjdk-devel wget
- wget https://downloads.apache.org/maven/maven-3/3.8.8/binaries/apache-maven-3.8.8-bin.tar.gz
- tar -xvf apache-maven-3.8.8-bin.tar.gz
- mv apache-maven-3.8.8 /usr/lib/maven
- - name: Build for Spark ${{ matrix.spark }}
+ - name: Build and run TPCH/DS
run: |
cd $GITHUB_WORKSPACE/
- export MAVEN_HOME=/usr/lib/maven
- export PATH=${PATH}:${MAVEN_HOME}/bin
- mvn clean install -P${{ matrix.spark }} -Pbackends-velox -DskipTests
- - name: Build and run TPCH/DS ${{ matrix.spark }}
- run: |
- cd $GITHUB_WORKSPACE/tools/gluten-it
- export MAVEN_HOME=/usr/lib/maven
- export PATH=${PATH}:${MAVEN_HOME}/bin
- mvn clean install -P${{ matrix.spark }} \
+ if [ "${{ matrix.java }}" = "java-17" ]; then
+ export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
+ else
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ fi
+ echo "JAVA_HOME: $JAVA_HOME"
+ mvn clean install -P${{ matrix.spark }} -P${{ matrix.java }} -Pbackends-velox -DskipTests
+ cd $GITHUB_WORKSPACE/tools/gluten-it
+ mvn clean install -P${{ matrix.spark }} -P${{ matrix.java }} \
&& GLUTEN_IT_JVM_ARGS=-Xmx5G sbin/gluten-it.sh queries-compare \
--local --preset=velox --benchmark-type=h --error-on-memleak --off-heap-size=10g -s=1.0 --threads=16 --iterations=1 \
&& GLUTEN_IT_JVM_ARGS=-Xmx5G sbin/gluten-it.sh queries-compare \
--local --preset=velox --benchmark-type=ds --error-on-memleak --off-heap-size=10g -s=1.0 --threads=16 --iterations=1
- run-tpc-test-centos8:
+ run-tpc-test-centos:
needs: build-native-lib
strategy:
fail-fast: false
matrix:
+ os: ["centos:7", "centos:8"]
spark: ["spark-3.2", "spark-3.3", "spark-3.4", "spark-3.5"]
+ java: ["java-8", "java-17"]
+ # Spark supports JDK17 since 3.3 and later, see https://issues.apache.org/jira/browse/SPARK-33772
+ exclude:
Review Comment:
Consider to use `include`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Support JDK17 [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5120:
URL: https://github.com/apache/incubator-gluten/pull/5120#issuecomment-2019885042
Run Gluten Clickhouse CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Support JDK17 [incubator-gluten]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on code in PR #5120:
URL: https://github.com/apache/incubator-gluten/pull/5120#discussion_r1538961770
##########
.github/workflows/velox_docker.yml:
##########
@@ -84,31 +89,41 @@ jobs:
path: ./cpp/build/releases
- name: Setup java and maven
run: |
- apt-get update && \
- apt-get install -y openjdk-8-jdk maven && \
+ if [[ "${{ matrix.java }}" == "17" ]]; then
+ apt-get update && \
+ apt-get install -y openjdk-17-jdk maven
+ else
+ apt-get update && \
+ apt-get install -y openjdk-8-jdk maven
+ fi
apt remove openjdk-11* -y
- name: Build for Spark ${{ matrix.spark }}
run: |
cd $GITHUB_WORKSPACE/ && \
- mvn clean install -P${{ matrix.spark }} -Pbackends-velox -DskipTests
+ mvn clean install -P${{ matrix.spark }} -P${{ matrix.java }} -Pbackends-velox -DskipTests
- name: Build and run TPCH/DS ${{ matrix.spark }}
run: |
cd $GITHUB_WORKSPACE/tools/gluten-it && \
- mvn clean install -P${{ matrix.spark }} \
+ mvn clean install -P${{ matrix.spark }} -P${{ matrix.java }} \
&& GLUTEN_IT_JVM_ARGS=-Xmx5G sbin/gluten-it.sh queries-compare \
--local --preset=velox --benchmark-type=h --error-on-memleak --off-heap-size=10g -s=1.0 --threads=16 --iterations=1 \
&& GLUTEN_IT_JVM_ARGS=-Xmx5G sbin/gluten-it.sh queries-compare \
--local --preset=velox --benchmark-type=ds --error-on-memleak --off-heap-size=10g -s=1.0 --threads=16 --iterations=1
-
- run-tpc-test-centos7:
+ run-tpc-test-centos:
needs: build-native-lib
strategy:
fail-fast: false
matrix:
+ os: ["centos:7", "centos:8"]
spark: ["spark-3.2", "spark-3.3", "spark-3.4", "spark-3.5"]
+ java: ["jdk-8", "jdk-17"]
Review Comment:
OK, in this pr I only add JDK17 test for Spark3.3. Do you have other concern ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Support JDK17 [incubator-gluten]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on PR #5120:
URL: https://github.com/apache/incubator-gluten/pull/5120#issuecomment-2173178261
@WangGuangxin I guess it's not related to JDK17, but https://github.com/facebookincubator/velox/discussions/9875 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org