You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/10/28 13:10:28 UTC
[GitHub] [spark] sarutak opened a new pull request #34425: [SPARK-37105][SQL][FOLLOWUP] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
sarutak opened a new pull request #34425:
URL: https://github.com/apache/spark/pull/34425
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
This PR fixes an issue that `HiveExternalCatalogVersionsSuite` fails with Java 17.
SPARK-37105 (#34376) seems to have fixed most of tests in `sql/hive` for Java 17 but `HiveExternalCatalogVersionsSuite`.
```
[info] org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite *** ABORTED *** (42 seconds, 526 milliseconds)
[info] spark-submit returned with exit code 1.
[info] Command line: '/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/bin/spark-submit' '--name' 'prepare testing tables' '--master' 'local[2]' '--conf' 'spark.ui.enabled=false' '--conf' 'spark.master.rest.enabled=false' '--conf' 'spark.sql.hive.metastore.version=2.3' '--conf' 'spark.sql.hive.metastore.jars=maven' '--conf' 'spark.sql.warehouse.dir=/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/warehouse-69d9bdbc-54ce-443b-8677-a413663ddb62' '--conf' 'spark.sql.test.version.index=0' '--driver-java-options' '-Dderby.system.home=/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/warehouse-69d9bdbc-54ce-443b-8677-a413663ddb62' '/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/tes
t15166225869206697603.py'
[info]
[info] 2021-10-28 06:07:18.486 - stderr> Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
[info] 2021-10-28 06:07:18.49 - stderr> 21/10/28 22:07:18 INFO SparkContext: Running Spark version 3.2.0
[info] 2021-10-28 06:07:18.537 - stderr> 21/10/28 22:07:18 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[info] 2021-10-28 06:07:18.616 - stderr> 21/10/28 22:07:18 INFO ResourceUtils: ==============================================================
[info] 2021-10-28 06:07:18.616 - stderr> 21/10/28 22:07:18 INFO ResourceUtils: No custom resources configured for spark.driver.
[info] 2021-10-28 06:07:18.616 - stderr> 21/10/28 22:07:18 INFO ResourceUtils: ==============================================================
[info] 2021-10-28 06:07:18.617 - stderr> 21/10/28 22:07:18 INFO SparkContext: Submitted application: prepare testing tables
[info] 2021-10-28 06:07:18.632 - stderr> 21/10/28 22:07:18 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
[info] 2021-10-28 06:07:18.641 - stderr> 21/10/28 22:07:18 INFO ResourceProfile: Limiting resource is cpu
[info] 2021-10-28 06:07:18.641 - stderr> 21/10/28 22:07:18 INFO ResourceProfileManager: Added ResourceProfile id: 0
[info] 2021-10-28 06:07:18.679 - stderr> 21/10/28 22:07:18 INFO SecurityManager: Changing view acls to: kou
[info] 2021-10-28 06:07:18.679 - stderr> 21/10/28 22:07:18 INFO SecurityManager: Changing modify acls to: kou
[info] 2021-10-28 06:07:18.68 - stderr> 21/10/28 22:07:18 INFO SecurityManager: Changing view acls groups to:
[info] 2021-10-28 06:07:18.68 - stderr> 21/10/28 22:07:18 INFO SecurityManager: Changing modify acls groups to:
[info] 2021-10-28 06:07:18.68 - stderr> 21/10/28 22:07:18 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(kou); groups with view permissions: Set(); users with modify permissions: Set(kou); groups with modify permissions: Set()
[info] 2021-10-28 06:07:18.886 - stderr> 21/10/28 22:07:18 INFO Utils: Successfully started service 'sparkDriver' on port 35867.
[info] 2021-10-28 06:07:18.906 - stderr> 21/10/28 22:07:18 INFO SparkEnv: Registering MapOutputTracker
[info] 2021-10-28 06:07:18.93 - stderr> 21/10/28 22:07:18 INFO SparkEnv: Registering BlockManagerMaster
[info] 2021-10-28 06:07:18.943 - stderr> 21/10/28 22:07:18 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
[info] 2021-10-28 06:07:18.944 - stderr> 21/10/28 22:07:18 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
[info] 2021-10-28 06:07:18.945 - stdout> Traceback (most recent call last):
[info] 2021-10-28 06:07:18.946 - stdout> File "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test15166225869206697603.py", line 5, in <module>
[info] 2021-10-28 06:07:18.946 - stdout> spark = SparkSession.builder.enableHiveSupport().getOrCreate()
[info] 2021-10-28 06:07:18.946 - stdout> File "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/python/lib/pyspark.zip/pyspark/sql/session.py", line 228, in getOrCreate
[info] 2021-10-28 06:07:18.946 - stdout> File "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/python/lib/pyspark.zip/pyspark/context.py", line 392, in getOrCreate
[info] 2021-10-28 06:07:18.946 - stdout> File "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/python/lib/pyspark.zip/pyspark/context.py", line 146, in __init__
[info] 2021-10-28 06:07:18.946 - stdout> File "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/python/lib/pyspark.zip/pyspark/context.py", line 209, in _do_init
[info] 2021-10-28 06:07:18.946 - stdout> File "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/python/lib/pyspark.zip/pyspark/context.py", line 329, in _initialize_context
[info] 2021-10-28 06:07:18.946 - stdout> File "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/python/lib/py4j-0.10.9.2-src.zip/py4j/java_gateway.py", line 1573, in __call__
[info] 2021-10-28 06:07:18.946 - stdout> File "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/python/lib/py4j-0.10.9.2-src.zip/py4j/protocol.py", line 326, in get_return_value
[info] 2021-10-28 06:07:18.947 - stdout> py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
[info] 2021-10-28 06:07:18.948 - stdout> : java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$ (in unnamed module @0x724b93a8) cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module @0x724b93a8
[info] 2021-10-28 06:07:18.948 - stdout> at org.apache.spark.storage.StorageUtils$.<init>(StorageUtils.scala:213)
[info] 2021-10-28 06:07:18.948 - stdout> at org.apache.spark.storage.StorageUtils$.<clinit>(StorageUtils.scala)
[info] 2021-10-28 06:07:18.948 - stdout> at org.apache.spark.storage.BlockManagerMasterEndpoint.<init>(BlockManagerMasterEndpoint.scala:110)
[info] 2021-10-28 06:07:18.948 - stdout> at org.apache.spark.SparkEnv$.$anonfun$create$9(SparkEnv.scala:348)
[info] 2021-10-28 06:07:18.948 - stdout> at org.apache.spark.SparkEnv$.registerOrLookupEndpoint$1(SparkEnv.scala:287)
[info] 2021-10-28 06:07:18.948 - stdout> at org.apache.spark.SparkEnv$.create(SparkEnv.scala:336)
[info] 2021-10-28 06:07:18.948 - stdout> at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:191)
[info] 2021-10-28 06:07:18.948 - stdout> at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:277)
[info] 2021-10-28 06:07:18.948 - stdout> at org.apache.spark.SparkContext.<init>(SparkContext.scala:460)
[info] 2021-10-28 06:07:18.948 - stdout> at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
[info] 2021-10-28 06:07:18.948 - stdout> at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[info] 2021-10-28 06:07:18.948 - stdout> at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
[info] 2021-10-28 06:07:18.948 - stdout> at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[info] 2021-10-28 06:07:18.948 - stdout> at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
[info] 2021-10-28 06:07:18.948 - stdout> at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
[info] 2021-10-28 06:07:18.948 - stdout> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
[info] 2021-10-28 06:07:18.948 - stdout> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
[info] 2021-10-28 06:07:18.948 - stdout> at py4j.Gateway.invoke(Gateway.java:238)
[info] 2021-10-28 06:07:18.948 - stdout> at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
[info] 2021-10-28 06:07:18.948 - stdout> at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
[info] 2021-10-28 06:07:18.948 - stdout> at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
[info] 2021-10-28 06:07:18.948 - stdout> at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
[info] 2021-10-28 06:07:18.948 - stdout> at java.base/java.lang.Thread.run(Thread.java:833)
[info] 2021-10-28 06:07:18.948 - stdout>
[info] 2021-10-28 06:07:18.975 - stderr> 21/10/28 22:07:18 INFO ShutdownHookManager: Shutdown hook called
[info] 2021-10-28 06:07:18.976 - stderr> 21/10/28 22:07:18 INFO ShutdownHookManager: Deleting directory /tmp/spark-5bd38557-2b71-43f1-80ee-3648a456b3fc (SparkSubmitTestUtils.scala:96)
[info] org.scalatest.exceptions.TestFailedException:
[info] at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
[info] at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
[info] at org.scalatest.funsuite.AnyFunSuite.newAssertionFailedException(AnyFunSuite.scala:1563)
[info] at org.scalatest.Assertions.fail(Assertions.scala:933)
[info] at org.scalatest.Assertions.fail$(Assertions.scala:929)
[info] at org.scalatest.funsuite.AnyFunSuite.fail(AnyFunSuite.scala:1563)
[info] at org.apache.spark.deploy.SparkSubmitTestUtils.runSparkSubmit(SparkSubmitTestUtils.scala:96)
[info] at org.apache.spark.deploy.SparkSubmitTestUtils.runSparkSubmit$(SparkSubmitTestUtils.scala:43)
[info] at org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.runSparkSubmit(HiveExternalCatalogVersionsSuite.scala:56)
[info] at org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.$anonfun$beforeAll$2(HiveExternalCatalogVersionsSuite.scala:223)
[info] at org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.$anonfun$beforeAll$2$adapted(HiveExternalCatalogVersionsSuite.scala:204)
[info] at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
[info] at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
[info] at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
[info] at org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.beforeAll(HiveExternalCatalogVersionsSuite.scala:204)
[info] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[info] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:62)
[info] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:318)
[info] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:513)
[info] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:413)
[info] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[info] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[info] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[info] at java.base/java.lang.Thread.run(Thread.java:833)
```
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
As we are trying to support Java 17, the suite should work with Java 17.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
No.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
Confirmed that the suite works with Java 17.
```
JAVA_HOME=/path/to/java17 build/sbt -Phive -Phive-thriftserver "testOnly org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #34425: [SPARK-37105][SQL][FOLLOWUP] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #34425:
URL: https://github.com/apache/spark/pull/34425#discussion_r738976571
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
##########
@@ -263,6 +267,8 @@ object PROCESS_TABLES extends QueryTest with SQLTestUtils {
versions
.filter(v => v.startsWith("3") || !TestUtils.isPythonVersionAtLeast38())
.filter(v => v.startsWith("3") || !SystemUtils.isJavaVersionAtLeast(JavaVersion.JAVA_9))
+ .filter(v => !((v.startsWith("3.0") || v.startsWith("3.1")) &&
+ SystemUtils.isJavaVersionAtLeast(JavaVersion.JAVA_17)))
Review comment:
Yes, we don't support it on 3.0 and 3.1 so they are filtered out if this test runs with Java 17.
Otherwise, we get the following error due to [this issue](https://github.com/scala/bug/issues/12419).
```
...
[info] 2021-10-28 23:49:34.926 - stderr> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[info] 2021-10-28 23:49:34.926 - stderr> at java.base/java.lang.reflect.Method.invoke(Method.java:568)
[info] 2021-10-28 23:49:34.926 - stderr> at java.base/java.lang.invoke.SerializedLambda.readResolve(SerializedLambda.java:278)
[info] 2021-10-28 23:49:34.926 - stderr> ... 33 more
[info] 2021-10-28 23:49:34.926 - stderr> Caused by: java.lang.IllegalArgumentException: too many arguments
[info] 2021-10-28 23:49:34.926 - stderr> at java.base/java.lang.invoke.LambdaMetafactory.altMetafactory(LambdaMetafactory.java:511)
[info] 2021-10-28 23:49:34.926 - stderr> at scala.runtime.LambdaDeserializer$.makeCallSite$1(LambdaDeserializer.scala:105)
[info] 2021-10-28 23:49:34.926 - stderr> at scala.runtime.LambdaDeserializer$.deserializeLambda(LambdaDeserializer.scala:114)
[info] 2021-10-28 23:49:34.926 - stderr> at scala.runtime.LambdaDeserialize.deserializeLambda(LambdaDeserialize.java:38)
[info] 2021-10-28 23:49:34.926 - stderr> at org.apache.spark.sql.execution.WholeStageCodegenExec.$deserializeLambda$(WholeStageCodegenExec.
...
```
If I misunderstand your comment, please let me know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954492641
**[Test build #144744 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144744/testReport)** for PR 34425 at commit [`09a73bf`](https://github.com/apache/spark/commit/09a73bf443aeabd81e0171a66f35e29f7e007ba2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954735991
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144751/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954695001
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49220/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34425: [SPARK-37105][SQL][FOLLOWUP] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-953942786
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49186/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954591700
**[Test build #144744 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144744/testReport)** for PR 34425 at commit [`09a73bf`](https://github.com/apache/spark/commit/09a73bf443aeabd81e0171a66f35e29f7e007ba2).
* This patch **fails SparkR unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak closed pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
sarutak closed pull request #34425:
URL: https://github.com/apache/spark/pull/34425
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-957031635
Merged to `master`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34425: [SPARK-37105][SQL][FOLLOWUP] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-953833926
**[Test build #144717 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144717/testReport)** for PR 34425 at commit [`8b92e76`](https://github.com/apache/spark/commit/8b92e7681eec4a1366517e76056c118ae9bfea29).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34425: [SPARK-37105][SQL][FOLLOWUP] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-953871305
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49186/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954492641
**[Test build #144744 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144744/testReport)** for PR 34425 at commit [`09a73bf`](https://github.com/apache/spark/commit/09a73bf443aeabd81e0171a66f35e29f7e007ba2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34425: [SPARK-37105][SQL][FOLLOWUP] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-953928044
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49186/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-955874606
**[Test build #144795 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144795/testReport)** for PR 34425 at commit [`e8f8fe7`](https://github.com/apache/spark/commit/e8f8fe76106f37fe05ca2bd94eb01b3fdb1eebfa).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-955865272
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-955921196
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144795/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954810315
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49226/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954831021
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144757/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954592542
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144744/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954583734
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34425:
URL: https://github.com/apache/spark/pull/34425#discussion_r739006444
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
##########
@@ -263,6 +267,8 @@ object PROCESS_TABLES extends QueryTest with SQLTestUtils {
versions
.filter(v => v.startsWith("3") || !TestUtils.isPythonVersionAtLeast38())
.filter(v => v.startsWith("3") || !SystemUtils.isJavaVersionAtLeast(JavaVersion.JAVA_9))
+ .filter(v => !((v.startsWith("3.0") || v.startsWith("3.1")) &&
+ SystemUtils.isJavaVersionAtLeast(JavaVersion.JAVA_17)))
Review comment:
> We don't support Java 17 on 3.0 and 3.1. I believe we can spin off this from this PR.
In addition, I didn't find that Spark 3.2.0 supports JDK 17 in [spark-release-3-2-0](http://spark.apache.org/releases/spark-release-3-2-0.html), so do we really need to test 3.2.0? Some pr for JDK 17 are not merge into 3.2, such as SPARK-37013
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954624192
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49214/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34425: [SPARK-37105][SQL][FOLLOWUP] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #34425:
URL: https://github.com/apache/spark/pull/34425#discussion_r738970408
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
##########
@@ -263,6 +267,8 @@ object PROCESS_TABLES extends QueryTest with SQLTestUtils {
versions
.filter(v => v.startsWith("3") || !TestUtils.isPythonVersionAtLeast38())
.filter(v => v.startsWith("3") || !SystemUtils.isJavaVersionAtLeast(JavaVersion.JAVA_9))
+ .filter(v => !((v.startsWith("3.0") || v.startsWith("3.1")) &&
+ SystemUtils.isJavaVersionAtLeast(JavaVersion.JAVA_17)))
Review comment:
We don't support Java 17 on 3.0 and 3.1. I believe we can spin off this from this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954560312
**[Test build #144741 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144741/testReport)** for PR 34425 at commit [`f592050`](https://github.com/apache/spark/commit/f592050e36bf6636ccfc3709f0d377c9dfe12602).
* This patch **fails SparkR unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954593052
The CI failure on Jenkins is not for the latest commit (729e05f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954624192
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49214/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954636736
**[Test build #144751 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144751/testReport)** for PR 34425 at commit [`729e05f`](https://github.com/apache/spark/commit/729e05fafdca5373b1a177ca5cd5dc3b2ba7d8e4).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34425: [SPARK-37105][SQL][FOLLOWUP] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-953833926
**[Test build #144717 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144717/testReport)** for PR 34425 at commit [`8b92e76`](https://github.com/apache/spark/commit/8b92e7681eec4a1366517e76056c118ae9bfea29).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #34425: [SPARK-37105][SQL][FOLLOWUP] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34425:
URL: https://github.com/apache/spark/pull/34425#discussion_r738894696
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
##########
@@ -218,7 +218,11 @@ class HiveExternalCatalogVersionsSuite extends SparkSubmitTestUtils {
"--conf", s"${HiveUtils.HIVE_METASTORE_JARS.key}=maven",
"--conf", s"${WAREHOUSE_PATH.key}=${wareHousePath.getCanonicalPath}",
"--conf", s"spark.sql.test.version.index=$index",
- "--driver-java-options", s"-Dderby.system.home=${wareHousePath.getCanonicalPath}",
+ "--driver-java-options", s"-Dderby.system.home=${wareHousePath.getCanonicalPath} " +
Review comment:
should we use `JavaModuleOptions.defaultModuleOptions` directly, although it contains some redundant args for this case, but it can be consistent with other places
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954614126
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49214/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954549043
**[Test build #144745 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144745/testReport)** for PR 34425 at commit [`729e05f`](https://github.com/apache/spark/commit/729e05fafdca5373b1a177ca5cd5dc3b2ba7d8e4).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954529874
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49210/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954541502
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49213/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #34425:
URL: https://github.com/apache/spark/pull/34425#discussion_r739029258
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
##########
@@ -218,7 +218,11 @@ class HiveExternalCatalogVersionsSuite extends SparkSubmitTestUtils {
"--conf", s"${HiveUtils.HIVE_METASTORE_JARS.key}=maven",
"--conf", s"${WAREHOUSE_PATH.key}=${wareHousePath.getCanonicalPath}",
"--conf", s"spark.sql.test.version.index=$index",
- "--driver-java-options", s"-Dderby.system.home=${wareHousePath.getCanonicalPath}",
+ "--driver-java-options", s"-Dderby.system.home=${wareHousePath.getCanonicalPath} " +
Review comment:
Oops, it's my mistake. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954633331
**[Test build #144745 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144745/testReport)** for PR 34425 at commit [`729e05f`](https://github.com/apache/spark/commit/729e05fafdca5373b1a177ca5cd5dc3b2ba7d8e4).
* This patch **fails SparkR unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954563856
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49210/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34425:
URL: https://github.com/apache/spark/pull/34425#discussion_r739008722
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
##########
@@ -263,6 +267,8 @@ object PROCESS_TABLES extends QueryTest with SQLTestUtils {
versions
.filter(v => v.startsWith("3") || !TestUtils.isPythonVersionAtLeast38())
.filter(v => v.startsWith("3") || !SystemUtils.isJavaVersionAtLeast(JavaVersion.JAVA_9))
+ .filter(v => !((v.startsWith("3.0") || v.startsWith("3.1")) &&
+ SystemUtils.isJavaVersionAtLeast(JavaVersion.JAVA_17)))
Review comment:
> Due to this issue, this test doesn't work with Java 17 for Spark 3.0 and Spark 3.1 which uses Scala < 2.12.15.
I have manually tested the new filter conditions. The version before filter includes `2.4.8,3.0.3,3.1.2,3.2.0` and the version after filter only includes `3.2.0`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954636736
**[Test build #144751 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144751/testReport)** for PR 34425 at commit [`729e05f`](https://github.com/apache/spark/commit/729e05fafdca5373b1a177ca5cd5dc3b2ba7d8e4).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954738059
**[Test build #144757 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144757/testReport)** for PR 34425 at commit [`e8f8fe7`](https://github.com/apache/spark/commit/e8f8fe76106f37fe05ca2bd94eb01b3fdb1eebfa).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954826062
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49226/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #34425: [SPARK-37105][SQL][FOLLOWUP] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34425:
URL: https://github.com/apache/spark/pull/34425#discussion_r738894696
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
##########
@@ -218,7 +218,11 @@ class HiveExternalCatalogVersionsSuite extends SparkSubmitTestUtils {
"--conf", s"${HiveUtils.HIVE_METASTORE_JARS.key}=maven",
"--conf", s"${WAREHOUSE_PATH.key}=${wareHousePath.getCanonicalPath}",
"--conf", s"spark.sql.test.version.index=$index",
- "--driver-java-options", s"-Dderby.system.home=${wareHousePath.getCanonicalPath}",
+ "--driver-java-options", s"-Dderby.system.home=${wareHousePath.getCanonicalPath} " +
Review comment:
should we use `JavaModuleOptions.defaultModuleOptions` directly, although it contains some redundant options for this case, but it can be consistent with other places
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-955874606
**[Test build #144795 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144795/testReport)** for PR 34425 at commit [`e8f8fe7`](https://github.com/apache/spark/commit/e8f8fe76106f37fe05ca2bd94eb01b3fdb1eebfa).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34425: [SPARK-37105][SQL][FOLLOWUP] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-953957520
**[Test build #144717 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144717/testReport)** for PR 34425 at commit [`8b92e76`](https://github.com/apache/spark/commit/8b92e7681eec4a1366517e76056c118ae9bfea29).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #34425: [SPARK-37105][SQL][FOLLOWUP] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #34425:
URL: https://github.com/apache/spark/pull/34425#discussion_r738971254
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
##########
@@ -218,7 +219,9 @@ class HiveExternalCatalogVersionsSuite extends SparkSubmitTestUtils {
"--conf", s"${HiveUtils.HIVE_METASTORE_JARS.key}=maven",
"--conf", s"${WAREHOUSE_PATH.key}=${wareHousePath.getCanonicalPath}",
"--conf", s"spark.sql.test.version.index=$index",
- "--driver-java-options", s"-Dderby.system.home=${wareHousePath.getCanonicalPath}",
+ "--driver-java-options", s"-Dderby.system.home=${wareHousePath.getCanonicalPath} " +
+ // TODO: Consider to remove the following JVM options once the Spark 3.2 line is EOL.
Review comment:
@sarutak . We had better use IDed TODO. Could you file a JIRA and make this `TODO(SPARK-XXX)` style?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-955909864
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49265/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-955918893
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49265/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak closed pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
sarutak closed pull request #34425:
URL: https://github.com/apache/spark/pull/34425
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-957031635
Merged to `master`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak edited a comment on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
sarutak edited a comment on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954593052
The SparkR unit test failure on Jenkins is not for the latest commit (729e05f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954770408
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49226/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954738059
**[Test build #144757 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144757/testReport)** for PR 34425 at commit [`e8f8fe7`](https://github.com/apache/spark/commit/e8f8fe76106f37fe05ca2bd94eb01b3fdb1eebfa).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #34425: [SPARK-37105][SQL][FOLLOWUP] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #34425:
URL: https://github.com/apache/spark/pull/34425#discussion_r738961289
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
##########
@@ -218,7 +218,11 @@ class HiveExternalCatalogVersionsSuite extends SparkSubmitTestUtils {
"--conf", s"${HiveUtils.HIVE_METASTORE_JARS.key}=maven",
"--conf", s"${WAREHOUSE_PATH.key}=${wareHousePath.getCanonicalPath}",
"--conf", s"spark.sql.test.version.index=$index",
- "--driver-java-options", s"-Dderby.system.home=${wareHousePath.getCanonicalPath}",
+ "--driver-java-options", s"-Dderby.system.home=${wareHousePath.getCanonicalPath} " +
Review comment:
Hmm, I just added the minimum required options but it's O.K to reuse the existing parts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954585150
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49214/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34425: [SPARK-37105][SQL][FOLLOWUP] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954467731
**[Test build #144741 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144741/testReport)** for PR 34425 at commit [`f592050`](https://github.com/apache/spark/commit/f592050e36bf6636ccfc3709f0d377c9dfe12602).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954634253
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144745/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954635275
retest this please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34425:
URL: https://github.com/apache/spark/pull/34425#discussion_r739044144
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
##########
@@ -263,6 +267,8 @@ object PROCESS_TABLES extends QueryTest with SQLTestUtils {
versions
.filter(v => v.startsWith("3") || !TestUtils.isPythonVersionAtLeast38())
.filter(v => v.startsWith("3") || !SystemUtils.isJavaVersionAtLeast(JavaVersion.JAVA_9))
+ .filter(v => !((v.startsWith("3.0") || v.startsWith("3.1")) &&
+ SystemUtils.isJavaVersionAtLeast(JavaVersion.JAVA_17)))
Review comment:
Manually test as follows:
```
mvn clean install -DskipTests -pl sql/hive -am -Phive-2.3 -Phive
mvn clean test -pl sql/hive -Phive-2.3 -Phive -Dtest=none -DwildcardSuites=org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite
```
```
Run completed in 1 minute, 28 seconds.
Total number of tests run: 1
Suites: completed 2, aborted 0
Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954686044
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49220/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954831021
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144757/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34425: [SPARK-37105][SQL][FOLLOWUP] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-953959400
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144717/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-955921196
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144795/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-955918893
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49265/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #34425: [SPARK-37105][SQL][FOLLOWUP] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #34425:
URL: https://github.com/apache/spark/pull/34425#discussion_r738976571
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
##########
@@ -263,6 +267,8 @@ object PROCESS_TABLES extends QueryTest with SQLTestUtils {
versions
.filter(v => v.startsWith("3") || !TestUtils.isPythonVersionAtLeast38())
.filter(v => v.startsWith("3") || !SystemUtils.isJavaVersionAtLeast(JavaVersion.JAVA_9))
+ .filter(v => !((v.startsWith("3.0") || v.startsWith("3.1")) &&
+ SystemUtils.isJavaVersionAtLeast(JavaVersion.JAVA_17)))
Review comment:
Yes, we don't support it on 3.0 and 3.1 so they are filtered out if this test runs with Java 17.
Otherwise, we get the following error due to [this issue](https://github.com/scala/bug/issues/12419).
If I misunderstand your comment, please let me know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #34425: [SPARK-37105][SQL][FOLLOWUP] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-953894142
Thank you very much. Let me verify this. It may be my negligence, because the environment I used to test cannot access internet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #34425: [SPARK-37105][SQL][FOLLOWUP] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #34425:
URL: https://github.com/apache/spark/pull/34425#discussion_r738377465
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
##########
@@ -263,6 +267,8 @@ object PROCESS_TABLES extends QueryTest with SQLTestUtils {
versions
.filter(v => v.startsWith("3") || !TestUtils.isPythonVersionAtLeast38())
.filter(v => v.startsWith("3") || !SystemUtils.isJavaVersionAtLeast(JavaVersion.JAVA_9))
+ .filter(v => !((v.startsWith("3.0") || v.startsWith("3.1")) &&
+ SystemUtils.isJavaVersionAtLeast(JavaVersion.JAVA_17)))
Review comment:
Due to [this issue](https://github.com/scala/scala/pull/9676), this test doesn't work with Java 17 for Spark 3.0 and Spark 3.1 which uses Scala `< 2.12.15`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #34425:
URL: https://github.com/apache/spark/pull/34425#discussion_r739012825
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
##########
@@ -263,6 +267,8 @@ object PROCESS_TABLES extends QueryTest with SQLTestUtils {
versions
.filter(v => v.startsWith("3") || !TestUtils.isPythonVersionAtLeast38())
.filter(v => v.startsWith("3") || !SystemUtils.isJavaVersionAtLeast(JavaVersion.JAVA_9))
+ .filter(v => !((v.startsWith("3.0") || v.startsWith("3.1")) &&
+ SystemUtils.isJavaVersionAtLeast(JavaVersion.JAVA_17)))
Review comment:
> I have manually tested the new filter conditions. The version before filter includes 2.4.8,3.0.3,3.1.2,3.2.0 and the version after filter only includes 3.2.0
It's intended. only `3.2.0` works with Java 17.
If it's not necessary to test with Java 17, I'll change this suite to just skip to run with Java 17.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34425:
URL: https://github.com/apache/spark/pull/34425#discussion_r739027309
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
##########
@@ -218,7 +218,11 @@ class HiveExternalCatalogVersionsSuite extends SparkSubmitTestUtils {
"--conf", s"${HiveUtils.HIVE_METASTORE_JARS.key}=maven",
"--conf", s"${WAREHOUSE_PATH.key}=${wareHousePath.getCanonicalPath}",
"--conf", s"spark.sql.test.version.index=$index",
- "--driver-java-options", s"-Dderby.system.home=${wareHousePath.getCanonicalPath}",
+ "--driver-java-options", s"-Dderby.system.home=${wareHousePath.getCanonicalPath} " +
Review comment:
Hmm, `JavaModuleOptions.defaultModuleOptions()` is already a String join with one space
, like
```
-XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED ...
```
Why we need to call `String.join` again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #34425:
URL: https://github.com/apache/spark/pull/34425#discussion_r739045783
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
##########
@@ -263,6 +267,8 @@ object PROCESS_TABLES extends QueryTest with SQLTestUtils {
versions
.filter(v => v.startsWith("3") || !TestUtils.isPythonVersionAtLeast38())
.filter(v => v.startsWith("3") || !SystemUtils.isJavaVersionAtLeast(JavaVersion.JAVA_9))
+ .filter(v => !((v.startsWith("3.0") || v.startsWith("3.1")) &&
+ SystemUtils.isJavaVersionAtLeast(JavaVersion.JAVA_17)))
Review comment:
The result is with this change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954695001
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49220/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-955920459
**[Test build #144795 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144795/testReport)** for PR 34425 at commit [`e8f8fe7`](https://github.com/apache/spark/commit/e8f8fe76106f37fe05ca2bd94eb01b3fdb1eebfa).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class SparkJobInfo(NamedTuple):`
* `class SparkStageInfo(NamedTuple):`
* `public class OrcColumnStatistics `
* `public class OrcFooterReader `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954583735
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954719454
Hmm, SparkR unit tests fail in another PR too.
https://github.com/apache/spark/pull/34426#issuecomment-954607423
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-955890966
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49265/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954661758
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49220/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak closed pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
sarutak closed pull request #34425:
URL: https://github.com/apache/spark/pull/34425
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34425: [SPARK-37105][SQL][FOLLOWUP] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-953942786
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49186/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34425: [SPARK-37105][SQL][FOLLOWUP] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-953959400
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144717/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #34425:
URL: https://github.com/apache/spark/pull/34425#discussion_r739012825
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
##########
@@ -263,6 +267,8 @@ object PROCESS_TABLES extends QueryTest with SQLTestUtils {
versions
.filter(v => v.startsWith("3") || !TestUtils.isPythonVersionAtLeast38())
.filter(v => v.startsWith("3") || !SystemUtils.isJavaVersionAtLeast(JavaVersion.JAVA_9))
+ .filter(v => !((v.startsWith("3.0") || v.startsWith("3.1")) &&
+ SystemUtils.isJavaVersionAtLeast(JavaVersion.JAVA_17)))
Review comment:
> I have manually tested the new filter conditions. The version before filter includes 2.4.8,3.0.3,3.1.2,3.2.0 and the version after filter only includes 3.2.0
It's intended. This test works only for `3.2.0` with Java 17.
If it's not necessary to test with Java 17, I'll change this suite to just skip to run with Java 17.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954592542
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144744/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954716192
**[Test build #144751 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144751/testReport)** for PR 34425 at commit [`729e05f`](https://github.com/apache/spark/commit/729e05fafdca5373b1a177ca5cd5dc3b2ba7d8e4).
* This patch **fails SparkR unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954735991
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144751/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954830042
**[Test build #144757 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144757/testReport)** for PR 34425 at commit [`e8f8fe7`](https://github.com/apache/spark/commit/e8f8fe76106f37fe05ca2bd94eb01b3fdb1eebfa).
* This patch **fails SparkR unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class SparkJobInfo(NamedTuple):`
* `class SparkStageInfo(NamedTuple):`
* `public class OrcColumnStatistics `
* `public class OrcFooterReader `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954826062
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49226/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954467731
**[Test build #144741 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144741/testReport)** for PR 34425 at commit [`f592050`](https://github.com/apache/spark/commit/f592050e36bf6636ccfc3709f0d377c9dfe12602).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #34425:
URL: https://github.com/apache/spark/pull/34425#discussion_r739045783
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
##########
@@ -263,6 +267,8 @@ object PROCESS_TABLES extends QueryTest with SQLTestUtils {
versions
.filter(v => v.startsWith("3") || !TestUtils.isPythonVersionAtLeast38())
.filter(v => v.startsWith("3") || !SystemUtils.isJavaVersionAtLeast(JavaVersion.JAVA_9))
+ .filter(v => !((v.startsWith("3.0") || v.startsWith("3.1")) &&
+ SystemUtils.isJavaVersionAtLeast(JavaVersion.JAVA_17)))
Review comment:
@LuciferYang Thank you for double-checking with Maven!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954572666
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49213/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954549043
**[Test build #144745 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144745/testReport)** for PR 34425 at commit [`729e05f`](https://github.com/apache/spark/commit/729e05fafdca5373b1a177ca5cd5dc3b2ba7d8e4).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-954634253
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144745/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #34425: [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #34425:
URL: https://github.com/apache/spark/pull/34425#issuecomment-957031635
Merged to `master`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org