You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by GitBox <gi...@apache.org> on 2022/09/18 03:26:52 UTC
[GitHub] [incubator-kyuubi] Yikf opened a new pull request, #3507: [KYUUBI #3441] Change default Spark version to 3.3.0
Yikf opened a new pull request, #3507:
URL: https://github.com/apache/incubator-kyuubi/pull/3507
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/CONTRIBUTING.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Fix https://github.com/apache/incubator-kyuubi/issues/3441, this pr aims to change default Spark version to 3.3.0
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] Yikf commented on pull request #3507: [KYUUBI #3441][WIP] Change default Spark version to 3.3.1
Posted by GitBox <gi...@apache.org>.
Yikf commented on PR #3507:
URL: https://github.com/apache/incubator-kyuubi/pull/3507#issuecomment-1260591356
@cfmcgrady Thanks, will test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] Yikf commented on a diff in pull request #3507: [KYUUBI #3441][WIP] Change default Spark version to 3.3.1
Posted by GitBox <gi...@apache.org>.
Yikf commented on code in PR #3507:
URL: https://github.com/apache/incubator-kyuubi/pull/3507#discussion_r973665608
##########
extensions/spark/kyuubi-spark-connector-tpcds/src/test/scala/org/apache/kyuubi/spark/connector/tpcds/TPCDSQuerySuite.scala:
##########
@@ -66,7 +66,7 @@ class TPCDSQuerySuite extends KyuubiFunSuite {
spark.sql("USE tpcds.tiny")
queries.map { queryName =>
val in = Utils.getContextOrKyuubiClassLoader
- .getResourceAsStream(s"kyuubi/tpcds_3.2/$queryName.sql")
+ .getResourceAsStream(s"kyuubi/tpcds_3.3/$queryName.sql")
Review Comment:
@pan3793 The form of output schemas is changed after https://github.com/apache/spark/pull/35227, So i re-generate golden files for Spark 3.3, Kyuubi does not currently run TPCDS tests for different Spark versions, but i think we should check for multiple versions, what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on pull request #3507: [KYUUBI #3441][WIP] Change default Spark version to 3.3.1
Posted by GitBox <gi...@apache.org>.
pan3793 commented on PR #3507:
URL: https://github.com/apache/incubator-kyuubi/pull/3507#issuecomment-1288104768
Spark 3.3.1 (RC4) vote passed.
https://www.mail-archive.com/dev@spark.apache.org/msg29565.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] Yikf commented on a diff in pull request #3507: [KYUUBI #3441][WIP] Change default Spark version to 3.3.1
Posted by GitBox <gi...@apache.org>.
Yikf commented on code in PR #3507:
URL: https://github.com/apache/incubator-kyuubi/pull/3507#discussion_r973676351
##########
extensions/spark/kyuubi-spark-connector-tpcds/src/test/scala/org/apache/kyuubi/spark/connector/tpcds/TPCDSQuerySuite.scala:
##########
@@ -66,7 +66,7 @@ class TPCDSQuerySuite extends KyuubiFunSuite {
spark.sql("USE tpcds.tiny")
queries.map { queryName =>
val in = Utils.getContextOrKyuubiClassLoader
- .getResourceAsStream(s"kyuubi/tpcds_3.2/$queryName.sql")
+ .getResourceAsStream(s"kyuubi/tpcds_3.3/$queryName.sql")
Review Comment:
> Does hash change as well?
TPC-H's hash changed also.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] Yikf commented on a diff in pull request #3507: [KYUUBI #3441][WIP] Change default Spark version to 3.3.1
Posted by GitBox <gi...@apache.org>.
Yikf commented on code in PR #3507:
URL: https://github.com/apache/incubator-kyuubi/pull/3507#discussion_r985105655
##########
.github/workflows/master.yml:
##########
@@ -385,10 +385,12 @@ jobs:
kubectl create serviceaccount spark
kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount=default:spark --namespace=default
kubectl get serviceaccount
+ # TODO Support more Spark version
- name: integration tests
run: >-
./build/mvn ${MVN_OPT} clean install
-Pflink-provided,hive-provided
+ -Pspark-3.2
Review Comment:
It looks like that we currently support [spark 3.2](https://github.com/apache/incubator-kyuubi/blob/master/integration-tests/kyuubi-kubernetes-it/src/test/scala/org/apache/kyuubi/kubernetes/test/spark/SparkOnKubernetesTestsSuite.scala#L48) only, We should specify the profile here to prevent some incompatibilities (on k8s w/ client mode)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] Yikf commented on a diff in pull request #3507: [KYUUBI #3441][WIP] Change default Spark version to 3.3.1
Posted by GitBox <gi...@apache.org>.
Yikf commented on code in PR #3507:
URL: https://github.com/apache/incubator-kyuubi/pull/3507#discussion_r991757789
##########
.github/workflows/master.yml:
##########
@@ -385,10 +385,12 @@ jobs:
kubectl create serviceaccount spark
kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount=default:spark --namespace=default
kubectl get serviceaccount
+ # TODO Support more Spark version
- name: integration tests
run: >-
./build/mvn ${MVN_OPT} clean install
-Pflink-provided,hive-provided
+ -Pspark-3.2
Review Comment:
In spark on k8s w/ client mode, task deserialization may fail due to inconsistent Jar packages on the Driver and Executor ends
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] cfmcgrady commented on pull request #3507: [KYUUBI #3441][WIP] Change default Spark version to 3.3.1
Posted by GitBox <gi...@apache.org>.
cfmcgrady commented on PR #3507:
URL: https://github.com/apache/incubator-kyuubi/pull/3507#issuecomment-1260583392
FYI: Spark-3.3.1-rc2 is out.
https://www.mail-archive.com/dev@spark.apache.org/msg29460.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] Yikf commented on pull request #3507: [KYUUBI #3441] Change default Spark version to 3.3.1
Posted by GitBox <gi...@apache.org>.
Yikf commented on PR #3507:
URL: https://github.com/apache/incubator-kyuubi/pull/3507#issuecomment-1290501043
Thanks all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] Yikf commented on pull request #3507: [KYUUBI #3441] Change default Spark version to 3.3.0
Posted by GitBox <gi...@apache.org>.
Yikf commented on PR #3507:
URL: https://github.com/apache/incubator-kyuubi/pull/3507#issuecomment-1250182971
@pan3793 , Tracking from here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] Yikf commented on a diff in pull request #3507: [KYUUBI #3441][WIP] Change default Spark version to 3.3.1
Posted by GitBox <gi...@apache.org>.
Yikf commented on code in PR #3507:
URL: https://github.com/apache/incubator-kyuubi/pull/3507#discussion_r991760580
##########
.github/workflows/master.yml:
##########
@@ -385,10 +385,12 @@ jobs:
kubectl create serviceaccount spark
kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount=default:spark --namespace=default
kubectl get serviceaccount
+ # TODO Support more Spark version
- name: integration tests
run: >-
./build/mvn ${MVN_OPT} clean install
-Pflink-provided,hive-provided
+ -Pspark-3.2
Review Comment:
Yes, I had the idea that we could pass the spark version of the current profile through the maven plugin so that the image version would be consistent with the spark version of the current profile, but I wanted to do this later for two reasons:
- This is to address the current todo and seems to be unrelated to the current pr
- The official spark image does not have a sub-version like Spark 3.3.1 https://hub.docker.com/r/apache/spark/tags, the Spark community is currently working on making the spark image the official docker image, maybe we can solve this todo after that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a diff in pull request #3507: [KYUUBI #3441][WIP] Change default Spark version to 3.3.1
Posted by GitBox <gi...@apache.org>.
pan3793 commented on code in PR #3507:
URL: https://github.com/apache/incubator-kyuubi/pull/3507#discussion_r991761300
##########
.github/workflows/master.yml:
##########
@@ -385,10 +385,12 @@ jobs:
kubectl create serviceaccount spark
kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount=default:spark --namespace=default
kubectl get serviceaccount
+ # TODO Support more Spark version
- name: integration tests
run: >-
./build/mvn ${MVN_OPT} clean install
-Pflink-provided,hive-provided
+ -Pspark-3.2
Review Comment:
make sense, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on pull request #3507: [KYUUBI #3441] Change default Spark version to 3.3.1
Posted by GitBox <gi...@apache.org>.
pan3793 commented on PR #3507:
URL: https://github.com/apache/incubator-kyuubi/pull/3507#issuecomment-1290457878
Thanks @Yikf, merging to master! Also thanks @wangyum for releasing Spark 3.3.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a diff in pull request #3507: [KYUUBI #3441][WIP] Change default Spark version to 3.3.1
Posted by GitBox <gi...@apache.org>.
pan3793 commented on code in PR #3507:
URL: https://github.com/apache/incubator-kyuubi/pull/3507#discussion_r991216171
##########
.github/workflows/master.yml:
##########
@@ -385,10 +385,12 @@ jobs:
kubectl create serviceaccount spark
kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount=default:spark --namespace=default
kubectl get serviceaccount
+ # TODO Support more Spark version
- name: integration tests
run: >-
./build/mvn ${MVN_OPT} clean install
-Pflink-provided,hive-provided
+ -Pspark-3.2
Review Comment:
it's tricky, what's the specific issue here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a diff in pull request #3507: [KYUUBI #3441][WIP] Change default Spark version to 3.3.1
Posted by GitBox <gi...@apache.org>.
pan3793 commented on code in PR #3507:
URL: https://github.com/apache/incubator-kyuubi/pull/3507#discussion_r973666634
##########
extensions/spark/kyuubi-spark-connector-tpcds/src/test/scala/org/apache/kyuubi/spark/connector/tpcds/TPCDSQuerySuite.scala:
##########
@@ -66,7 +66,7 @@ class TPCDSQuerySuite extends KyuubiFunSuite {
spark.sql("USE tpcds.tiny")
queries.map { queryName =>
val in = Utils.getContextOrKyuubiClassLoader
- .getResourceAsStream(s"kyuubi/tpcds_3.2/$queryName.sql")
+ .getResourceAsStream(s"kyuubi/tpcds_3.3/$queryName.sql")
Review Comment:
Does hash change as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] Yikf commented on a diff in pull request #3507: [KYUUBI #3441][WIP] Change default Spark version to 3.3.1
Posted by GitBox <gi...@apache.org>.
Yikf commented on code in PR #3507:
URL: https://github.com/apache/incubator-kyuubi/pull/3507#discussion_r973666486
##########
extensions/spark/kyuubi-spark-connector-tpcds/src/test/scala/org/apache/kyuubi/spark/connector/tpcds/TPCDSQuerySuite.scala:
##########
@@ -66,7 +66,7 @@ class TPCDSQuerySuite extends KyuubiFunSuite {
spark.sql("USE tpcds.tiny")
queries.map { queryName =>
val in = Utils.getContextOrKyuubiClassLoader
- .getResourceAsStream(s"kyuubi/tpcds_3.2/$queryName.sql")
+ .getResourceAsStream(s"kyuubi/tpcds_3.3/$queryName.sql")
Review Comment:
~~emm, I mean should we check mutiple Spark versions? In the tpcds_3.2 directory, multiple directories have output schema and hash output for each Spark version~~
```
- tpcds_3.2
q1.sql ---file
... ---file
- spark 3.1 out ---dir
- spark 3.2 out ---dir
- spark 3.3 out ---dir
```
> Update schema to match the lastest Spark version is fine
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a diff in pull request #3507: [KYUUBI #3441][WIP] Change default Spark version to 3.3.1
Posted by GitBox <gi...@apache.org>.
pan3793 commented on code in PR #3507:
URL: https://github.com/apache/incubator-kyuubi/pull/3507#discussion_r973665793
##########
extensions/spark/kyuubi-spark-connector-tpcds/src/test/scala/org/apache/kyuubi/spark/connector/tpcds/TPCDSQuerySuite.scala:
##########
@@ -66,7 +66,7 @@ class TPCDSQuerySuite extends KyuubiFunSuite {
spark.sql("USE tpcds.tiny")
queries.map { queryName =>
val in = Utils.getContextOrKyuubiClassLoader
- .getResourceAsStream(s"kyuubi/tpcds_3.2/$queryName.sql")
+ .getResourceAsStream(s"kyuubi/tpcds_3.3/$queryName.sql")
Review Comment:
"3.2" is not Spark version, it's TPC-DS version
https://tpc.org/TPC_Documents_Current_Versions/pdf/TPC-DS_v3.2.0.pdf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a diff in pull request #3507: [KYUUBI #3441][WIP] Change default Spark version to 3.3.1
Posted by GitBox <gi...@apache.org>.
pan3793 commented on code in PR #3507:
URL: https://github.com/apache/incubator-kyuubi/pull/3507#discussion_r973665877
##########
extensions/spark/kyuubi-spark-connector-tpcds/src/test/scala/org/apache/kyuubi/spark/connector/tpcds/TPCDSQuerySuite.scala:
##########
@@ -66,7 +66,7 @@ class TPCDSQuerySuite extends KyuubiFunSuite {
spark.sql("USE tpcds.tiny")
queries.map { queryName =>
val in = Utils.getContextOrKyuubiClassLoader
- .getResourceAsStream(s"kyuubi/tpcds_3.2/$queryName.sql")
+ .getResourceAsStream(s"kyuubi/tpcds_3.3/$queryName.sql")
Review Comment:
Update schema to match the lastest Spark version is fine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a diff in pull request #3507: [KYUUBI #3441][WIP] Change default Spark version to 3.3.1
Posted by GitBox <gi...@apache.org>.
pan3793 commented on code in PR #3507:
URL: https://github.com/apache/incubator-kyuubi/pull/3507#discussion_r991758711
##########
.github/workflows/master.yml:
##########
@@ -385,10 +385,12 @@ jobs:
kubectl create serviceaccount spark
kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount=default:spark --namespace=default
kubectl get serviceaccount
+ # TODO Support more Spark version
- name: integration tests
run: >-
./build/mvn ${MVN_OPT} clean install
-Pflink-provided,hive-provided
+ -Pspark-3.2
Review Comment:
oh sorry, i missed we hardcoded the spark image in code, how about changing it to get from ENV then we can override the in test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 closed pull request #3507: [KYUUBI #3441] Change default Spark version to 3.3.1
Posted by GitBox <gi...@apache.org>.
pan3793 closed pull request #3507: [KYUUBI #3441] Change default Spark version to 3.3.1
URL: https://github.com/apache/incubator-kyuubi/pull/3507
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a diff in pull request #3507: [KYUUBI #3441][WIP] Change default Spark version to 3.3.1
Posted by GitBox <gi...@apache.org>.
pan3793 commented on code in PR #3507:
URL: https://github.com/apache/incubator-kyuubi/pull/3507#discussion_r991759777
##########
.github/workflows/master.yml:
##########
@@ -385,10 +385,12 @@ jobs:
kubectl create serviceaccount spark
kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount=default:spark --namespace=default
kubectl get serviceaccount
+ # TODO Support more Spark version
- name: integration tests
run: >-
./build/mvn ${MVN_OPT} clean install
-Pflink-provided,hive-provided
+ -Pspark-3.2
Review Comment:
We can get the compile Spark version by
```
SPARK_VERSION=$("$MVN" help:evaluate -Dexpression=spark.version $@ 2>/dev/null\
| grep -v "INFO"\
| grep -v "WARNING"\
| tail -n 1)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] Yikf commented on pull request #3507: [KYUUBI #3441][WIP] Change default Spark version to 3.3.1
Posted by GitBox <gi...@apache.org>.
Yikf commented on PR #3507:
URL: https://github.com/apache/incubator-kyuubi/pull/3507#issuecomment-1264204648
> The test failed consistently, could you please take a look?
will check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] Yikf commented on a diff in pull request #3507: [KYUUBI #3441][WIP] Change default Spark version to 3.3.1
Posted by GitBox <gi...@apache.org>.
Yikf commented on code in PR #3507:
URL: https://github.com/apache/incubator-kyuubi/pull/3507#discussion_r973667190
##########
extensions/spark/kyuubi-spark-connector-tpcds/src/test/scala/org/apache/kyuubi/spark/connector/tpcds/TPCDSQuerySuite.scala:
##########
@@ -66,7 +66,7 @@ class TPCDSQuerySuite extends KyuubiFunSuite {
spark.sql("USE tpcds.tiny")
queries.map { queryName =>
val in = Utils.getContextOrKyuubiClassLoader
- .getResourceAsStream(s"kyuubi/tpcds_3.2/$queryName.sql")
+ .getResourceAsStream(s"kyuubi/tpcds_3.3/$queryName.sql")
Review Comment:
It look like no change, wait CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] Yikf commented on a diff in pull request #3507: [KYUUBI #3441][WIP] Change default Spark version to 3.3.1
Posted by GitBox <gi...@apache.org>.
Yikf commented on code in PR #3507:
URL: https://github.com/apache/incubator-kyuubi/pull/3507#discussion_r973666486
##########
extensions/spark/kyuubi-spark-connector-tpcds/src/test/scala/org/apache/kyuubi/spark/connector/tpcds/TPCDSQuerySuite.scala:
##########
@@ -66,7 +66,7 @@ class TPCDSQuerySuite extends KyuubiFunSuite {
spark.sql("USE tpcds.tiny")
queries.map { queryName =>
val in = Utils.getContextOrKyuubiClassLoader
- .getResourceAsStream(s"kyuubi/tpcds_3.2/$queryName.sql")
+ .getResourceAsStream(s"kyuubi/tpcds_3.3/$queryName.sql")
Review Comment:
emm, I mean should we check mutiple Spark versions? In the tpcds_3.2 directory, multiple directories have output schema and hash output for each Spark version
```
- tpcds_3.2
q1.sql ---file
... ---file
- spark 3.1 out ---dir
- spark 3.2 out ---dir
- spark 3.3 out ---dir
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on pull request #3507: [KYUUBI #3441][WIP] Change default Spark version to 3.3.1
Posted by GitBox <gi...@apache.org>.
pan3793 commented on PR #3507:
URL: https://github.com/apache/incubator-kyuubi/pull/3507#issuecomment-1263497699
The test failed consistently, could you please take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] Yikf commented on pull request #3507: [KYUUBI #3441][WIP] Change default Spark version to 3.3.1
Posted by GitBox <gi...@apache.org>.
Yikf commented on PR #3507:
URL: https://github.com/apache/incubator-kyuubi/pull/3507#issuecomment-1280183418
Test for Spark 3.3.1 (RC4)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] Yikf commented on a diff in pull request #3507: [KYUUBI #3441][WIP] Change default Spark version to 3.3.1
Posted by GitBox <gi...@apache.org>.
Yikf commented on code in PR #3507:
URL: https://github.com/apache/incubator-kyuubi/pull/3507#discussion_r985105655
##########
.github/workflows/master.yml:
##########
@@ -385,10 +385,12 @@ jobs:
kubectl create serviceaccount spark
kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount=default:spark --namespace=default
kubectl get serviceaccount
+ # TODO Support more Spark version
- name: integration tests
run: >-
./build/mvn ${MVN_OPT} clean install
-Pflink-provided,hive-provided
+ -Pspark-3.2
Review Comment:
It looks like that we currently support [spark 3.2](https://github.com/apache/incubator-kyuubi/blob/master/integration-tests/kyuubi-kubernetes-it/src/test/scala/org/apache/kyuubi/kubernetes/test/spark/SparkOnKubernetesTestsSuite.scala#L48) only, We should specify the profile here to prevent some incompatibilities (on k8s w/ client)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org