You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "zery (Jira)" <ji...@apache.org> on 2022/04/07 09:30:00 UTC
[jira] [Created] (SPARK-38815) Not found jdbc driver class when enable hive and call multiple jdbc action
zery created SPARK-38815:
----------------------------
Summary: Not found jdbc driver class when enable hive and call multiple jdbc action
Key: SPARK-38815
URL: https://issues.apache.org/jira/browse/SPARK-38815
Project: Spark
Issue Type: Bug
Components: Kubernetes, Spark Core, SQL
Affects Versions: 3.1.1
Environment: k8s: v1.20.4
spark: v3.1.1
hive: 4.0.0-SNAPSHOT
Reporter: zery
Hello, the spark code is:
{code:java}
// Some comments here
def main(args : Array[String]): Unit = {
val spark = SparkSession.builder
.appName("TestCKJdbc")
.enableHiveSupport()
.getOrCreate()
spark.read
.format("jdbc")
.option("driver","ru.yandex.clickhouse.ClickHouseDriver")
.option("url", "jdbc:clickhouse://clickhouse-server-svc.admin.svc.cluster.local:8123/aaa")
.option("dbtable", "A")
.option("user", "default")
.option("password", "abc")
.load()
.show()
spark.read
.format("jdbc")
.option("driver","ru.yandex.clickhouse.ClickHouseDriver")
.option("url", "jdbc:clickhouse://clickhouse-server-svc.admin.svc.cluster.local:8123/aaa")
.option("dbtable", "A")
.option("user", "default")
.option("password", "abc")
.load()
.show()
spark.stop()
}
{code}
when I submit it to k8s, it will get error:
{code:java}
22/04/07 09:12:08 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0) (10.233.67.129, executor 1, partition 0, PROCESS_LOCAL, 4318 bytes) taskResourceAssignments Map()
22/04/07 09:12:09 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.233.67.129:41816 (size: 5.1 KiB, free: 413.9 MiB)
22/04/07 09:12:15 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 6596 ms on 10.233.67.129 (executor 1) (1/1)
22/04/07 09:12:15 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
22/04/07 09:12:15 INFO DAGScheduler: ResultStage 0 (show at App.scala:24) finished in 8.524 s
22/04/07 09:12:15 INFO DAGScheduler: Job 0 is finished. Cancelling potential speculative or zombie tasks for this job
22/04/07 09:12:15 INFO TaskSchedulerImpl: Killing all running tasks in stage 0: Stage finished
22/04/07 09:12:15 INFO DAGScheduler: Job 0 finished: show at App.scala:24, took 8.611591 s
22/04/07 09:12:15 INFO CodeGenerator: Code generated in 47.462472 ms
+---+---------+---------+
| id|discrete1|discrete2|
+---+---------+---------+
| 1| A| a|
| 1| B| b|
| 1| A| c|
| 2| C| a|
| 1| A| a|
+---+---------+---------+22/04/07 09:12:15 INFO SparkUI: Stopped Spark web UI at http://spark-7f8c3c80034ac34e-driver-svc.spark-operator.svc:4040
22/04/07 09:12:15 INFO KubernetesClusterSchedulerBackend: Shutting down all executors
22/04/07 09:12:15 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking each executor to shut down
22/04/07 09:12:15 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed (this is expected if the application is shutting down.)
22/04/07 09:12:15 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
22/04/07 09:12:15 INFO MemoryStore: MemoryStore cleared
22/04/07 09:12:15 INFO BlockManager: BlockManager stopped
22/04/07 09:12:15 INFO BlockManagerMaster: BlockManagerMaster stopped
22/04/07 09:12:15 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
22/04/07 09:12:15 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main" java.lang.ClassNotFoundException: ru.yandex.clickhouse.ClickHouseDriver
at java.base/java.net.URLClassLoader.findClass(Unknown Source)
at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
at org.apache.spark.sql.execution.datasources.jdbc.DriverRegistry$.register(DriverRegistry.scala:46)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.$anonfun$driverClass$1(JDBCOptions.scala:102)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.$anonfun$driverClass$1$adapted(JDBCOptions.scala:102)
at scala.Option.foreach(Option.scala:407)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.<init>(JDBCOptions.scala:102)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.<init>(JDBCOptions.scala:38)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:32)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:354)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:326)
at org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:308)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:308)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:226)
at org.example.App$.main(App.scala:34)
at org.example.App.main(App.scala)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1039)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1048)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
22/04/07 09:12:15 INFO ShutdownHookManager: Shutdown hook called
22/04/07 09:12:15 INFO ShutdownHookManager: Deleting directory /tmp/spark-77261146-995b-42e6-b5e8-eb7cbf8eda49
22/04/07 09:12:15 INFO ShutdownHookManager: Deleting directory /var/data/spark-5afa86cb-67c6-493a-90f5-ea1c23089c9f/spark-d1c96f2e-c40c-4942-bd99-0451d23159b3
22/04/07 09:12:15 INFO AuditProviderFactory: ==> JVMShutdownHook.run()
22/04/07 09:12:15 INFO AuditProviderFactory: JVMShutdownHook: Signalling async audit cleanup to start.
22/04/07 09:12:15 INFO AuditProviderFactory: JVMShutdownHook: Waiting up to 30 seconds for audit cleanup to finish.
22/04/07 09:12:15 INFO AuditProviderFactory: RangerAsyncAuditCleanup: Starting cleanup
22/04/07 09:12:15 INFO AuditAsyncQueue: Stop called. name=hiveCLI.async
22/04/07 09:12:15 INFO AuditAsyncQueue: Interrupting consumerThread. name=hiveCLI.async, consumer=hiveCLI.async.batch
22/04/07 09:12:15 INFO AuditProviderFactory: RangerAsyncAuditCleanup: Done cleanup {code}
**but, if del ".enableHiveSupport()" code, it will run success.*
like:
{code:java}
def main(args : Array[String]): Unit = {
val spark = SparkSession.builder
.appName("TestCKJdbc")
//.enableHiveSupport()
.getOrCreate()
spark.read
.format("jdbc")
.option("driver","ru.yandex.clickhouse.ClickHouseDriver")
.option("url", "jdbc:clickhouse://clickhouse-server-svc.admin.svc.cluster.local:8123/abc")
.option("dbtable", "A")
.option("user", "default")
.option("password", "aaa")
.load()
.show()
spark.read
.format("jdbc")
.option("driver","ru.yandex.clickhouse.ClickHouseDriver")
.option("url", "jdbc:clickhouse://clickhouse-server-svc.admin.svc.cluster.local:8123/abc")
.option("dbtable", "A")
.option("user", "default")
.option("password", "aaa")
.load()
.show()
spark.stop()
}
{code}
It will happen only when enableHive and call multiple jdbc action function.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org