You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2021/10/29 07:22:00 UTC
[jira] [Assigned] (SPARK-37159) Change
HiveExternalCatalogVersionsSuite to be able to test with Java 17
[ https://issues.apache.org/jira/browse/SPARK-37159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-37159:
------------------------------------
Assignee: Apache Spark (was: Kousuke Saruta)
> Change HiveExternalCatalogVersionsSuite to be able to test with Java 17
> -----------------------------------------------------------------------
>
> Key: SPARK-37159
> URL: https://issues.apache.org/jira/browse/SPARK-37159
> Project: Spark
> Issue Type: Bug
> Components: SQL, Tests
> Affects Versions: 3.3.0
> Reporter: Kousuke Saruta
> Assignee: Apache Spark
> Priority: Minor
>
> SPARK-37105 seems to have fixed most of tests in `sql/hive` for Java 17 but `HiveExternalCatalogVersionsSuite`.
> {code}
> [info] org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite *** ABORTED *** (42 seconds, 526 milliseconds)
> [info] spark-submit returned with exit code 1.
> [info] Command line: '/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/bin/spark-submit' '--name' 'prepare testing tables' '--master' 'local[2]' '--conf' 'spark.ui.enabled=false' '--conf' 'spark.master.rest.enabled=false' '--conf' 'spark.sql.hive.metastore.version=2.3' '--conf' 'spark.sql.hive.metastore.jars=maven' '--conf' 'spark.sql.warehouse.dir=/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/warehouse-69d9bdbc-54ce-443b-8677-a413663ddb62' '--conf' 'spark.sql.test.version.index=0' '--driver-java-options' '-Dderby.system.home=/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/warehouse-69d9bdbc-54ce-443b-8677-a413663ddb62' '/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test15166225869206697603.py'
> [info]
> [info] 2021-10-28 06:07:18.486 - stderr> Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
> [info] 2021-10-28 06:07:18.49 - stderr> 21/10/28 22:07:18 INFO SparkContext: Running Spark version 3.2.0
> [info] 2021-10-28 06:07:18.537 - stderr> 21/10/28 22:07:18 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> [info] 2021-10-28 06:07:18.616 - stderr> 21/10/28 22:07:18 INFO ResourceUtils: ==============================================================
> [info] 2021-10-28 06:07:18.616 - stderr> 21/10/28 22:07:18 INFO ResourceUtils: No custom resources configured for spark.driver.
> [info] 2021-10-28 06:07:18.616 - stderr> 21/10/28 22:07:18 INFO ResourceUtils: ==============================================================
> [info] 2021-10-28 06:07:18.617 - stderr> 21/10/28 22:07:18 INFO SparkContext: Submitted application: prepare testing tables
> [info] 2021-10-28 06:07:18.632 - stderr> 21/10/28 22:07:18 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
> [info] 2021-10-28 06:07:18.641 - stderr> 21/10/28 22:07:18 INFO ResourceProfile: Limiting resource is cpu
> [info] 2021-10-28 06:07:18.641 - stderr> 21/10/28 22:07:18 INFO ResourceProfileManager: Added ResourceProfile id: 0
> [info] 2021-10-28 06:07:18.679 - stderr> 21/10/28 22:07:18 INFO SecurityManager: Changing view acls to: kou
> [info] 2021-10-28 06:07:18.679 - stderr> 21/10/28 22:07:18 INFO SecurityManager: Changing modify acls to: kou
> [info] 2021-10-28 06:07:18.68 - stderr> 21/10/28 22:07:18 INFO SecurityManager: Changing view acls groups to:
> [info] 2021-10-28 06:07:18.68 - stderr> 21/10/28 22:07:18 INFO SecurityManager: Changing modify acls groups to:
> [info] 2021-10-28 06:07:18.68 - stderr> 21/10/28 22:07:18 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(kou); groups with view permissions: Set(); users with modify permissions: Set(kou); groups with modify permissions: Set()
> [info] 2021-10-28 06:07:18.886 - stderr> 21/10/28 22:07:18 INFO Utils: Successfully started service 'sparkDriver' on port 35867.
> [info] 2021-10-28 06:07:18.906 - stderr> 21/10/28 22:07:18 INFO SparkEnv: Registering MapOutputTracker
> [info] 2021-10-28 06:07:18.93 - stderr> 21/10/28 22:07:18 INFO SparkEnv: Registering BlockManagerMaster
> [info] 2021-10-28 06:07:18.943 - stderr> 21/10/28 22:07:18 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
> [info] 2021-10-28 06:07:18.944 - stderr> 21/10/28 22:07:18 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
> [info] 2021-10-28 06:07:18.945 - stdout> Traceback (most recent call last):
> [info] 2021-10-28 06:07:18.946 - stdout> File "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test15166225869206697603.py", line 5, in <module>
> [info] 2021-10-28 06:07:18.946 - stdout> spark = SparkSession.builder.enableHiveSupport().getOrCreate()
> [info] 2021-10-28 06:07:18.946 - stdout> File "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/python/lib/pyspark.zip/pyspark/sql/session.py", line 228, in getOrCreate
> [info] 2021-10-28 06:07:18.946 - stdout> File "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/python/lib/pyspark.zip/pyspark/context.py", line 392, in getOrCreate
> [info] 2021-10-28 06:07:18.946 - stdout> File "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/python/lib/pyspark.zip/pyspark/context.py", line 146, in __init__
> [info] 2021-10-28 06:07:18.946 - stdout> File "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/python/lib/pyspark.zip/pyspark/context.py", line 209, in _do_init
> [info] 2021-10-28 06:07:18.946 - stdout> File "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/python/lib/pyspark.zip/pyspark/context.py", line 329, in _initialize_context
> [info] 2021-10-28 06:07:18.946 - stdout> File "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/python/lib/py4j-0.10.9.2-src.zip/py4j/java_gateway.py", line 1573, in __call__
> [info] 2021-10-28 06:07:18.946 - stdout> File "/home/kou/work/oss/spark-java17/sql/hive/target/tmp/org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite/test-spark-d86af275-0c40-4b47-9cab-defa92a5ffa7/spark-3.2.0/python/lib/py4j-0.10.9.2-src.zip/py4j/protocol.py", line 326, in get_return_value
> [info] 2021-10-28 06:07:18.947 - stdout> py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
> [info] 2021-10-28 06:07:18.948 - stdout> : java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$ (in unnamed module @0x724b93a8) cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module @0x724b93a8
> [info] 2021-10-28 06:07:18.948 - stdout> at org.apache.spark.storage.StorageUtils$.<init>(StorageUtils.scala:213)
> [info] 2021-10-28 06:07:18.948 - stdout> at org.apache.spark.storage.StorageUtils$.<clinit>(StorageUtils.scala)
> [info] 2021-10-28 06:07:18.948 - stdout> at org.apache.spark.storage.BlockManagerMasterEndpoint.<init>(BlockManagerMasterEndpoint.scala:110)
> [info] 2021-10-28 06:07:18.948 - stdout> at org.apache.spark.SparkEnv$.$anonfun$create$9(SparkEnv.scala:348)
> [info] 2021-10-28 06:07:18.948 - stdout> at org.apache.spark.SparkEnv$.registerOrLookupEndpoint$1(SparkEnv.scala:287)
> [info] 2021-10-28 06:07:18.948 - stdout> at org.apache.spark.SparkEnv$.create(SparkEnv.scala:336)
> [info] 2021-10-28 06:07:18.948 - stdout> at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:191)
> [info] 2021-10-28 06:07:18.948 - stdout> at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:277)
> [info] 2021-10-28 06:07:18.948 - stdout> at org.apache.spark.SparkContext.<init>(SparkContext.scala:460)
> [info] 2021-10-28 06:07:18.948 - stdout> at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
> [info] 2021-10-28 06:07:18.948 - stdout> at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> [info] 2021-10-28 06:07:18.948 - stdout> at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
> [info] 2021-10-28 06:07:18.948 - stdout> at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> [info] 2021-10-28 06:07:18.948 - stdout> at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
> [info] 2021-10-28 06:07:18.948 - stdout> at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
> [info] 2021-10-28 06:07:18.948 - stdout> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
> [info] 2021-10-28 06:07:18.948 - stdout> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
> [info] 2021-10-28 06:07:18.948 - stdout> at py4j.Gateway.invoke(Gateway.java:238)
> [info] 2021-10-28 06:07:18.948 - stdout> at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
> [info] 2021-10-28 06:07:18.948 - stdout> at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
> [info] 2021-10-28 06:07:18.948 - stdout> at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
> [info] 2021-10-28 06:07:18.948 - stdout> at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
> [info] 2021-10-28 06:07:18.948 - stdout> at java.base/java.lang.Thread.run(Thread.java:833)
> [info] 2021-10-28 06:07:18.948 - stdout>
> [info] 2021-10-28 06:07:18.975 - stderr> 21/10/28 22:07:18 INFO ShutdownHookManager: Shutdown hook called
> [info] 2021-10-28 06:07:18.976 - stderr> 21/10/28 22:07:18 INFO ShutdownHookManager: Deleting directory /tmp/spark-5bd38557-2b71-43f1-80ee-3648a456b3fc (SparkSubmitTestUtils.scala:96)
> [info] org.scalatest.exceptions.TestFailedException:
> [info] at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
> [info] at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
> [info] at org.scalatest.funsuite.AnyFunSuite.newAssertionFailedException(AnyFunSuite.scala:1563)
> [info] at org.scalatest.Assertions.fail(Assertions.scala:933)
> [info] at org.scalatest.Assertions.fail$(Assertions.scala:929)
> [info] at org.scalatest.funsuite.AnyFunSuite.fail(AnyFunSuite.scala:1563)
> [info] at org.apache.spark.deploy.SparkSubmitTestUtils.runSparkSubmit(SparkSubmitTestUtils.scala:96)
> [info] at org.apache.spark.deploy.SparkSubmitTestUtils.runSparkSubmit$(SparkSubmitTestUtils.scala:43)
> [info] at org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.runSparkSubmit(HiveExternalCatalogVersionsSuite.scala:56)
> [info] at org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.$anonfun$beforeAll$2(HiveExternalCatalogVersionsSuite.scala:223)
> [info] at org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.$anonfun$beforeAll$2$adapted(HiveExternalCatalogVersionsSuite.scala:204)
> [info] at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> [info] at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> [info] at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
> [info] at org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite.beforeAll(HiveExternalCatalogVersionsSuite.scala:204)
> [info] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
> [info] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
> [info] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
> [info] at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:62)
> [info] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:318)
> [info] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:513)
> [info] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:413)
> [info] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> [info] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> [info] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> [info] at java.base/java.lang.Thread.run(Thread.java:833)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org