You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by SlavikBaranov <gi...@git.apache.org> on 2015/04/29 16:58:09 UTC
[GitHub] spark pull request: [SPARK-6913] Fixed "java.sql.SQLException: No ...
GitHub user SlavikBaranov opened a pull request:
https://github.com/apache/spark/pull/5782
[SPARK-6913] Fixed "java.sql.SQLException: No suitable driver found"
Fixed `java.sql.SQLException: No suitable driver found` when loading DataFrame into Spark SQL if the driver is supplied with `--jars` argument.
The problem is in `java.sql.DriverManager` class that can't access drivers loaded by Spark ClassLoader.
Wrappers that forward requests are created for these drivers.
Also, it's not necessary any more to include JDBC drivers in `--driver-class-path` in local mode, specifying in `--jars` argument is sufficient.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/SlavikBaranov/spark SPARK-6913
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5782.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5782
----
commit c8294aea77daa51090f3403a5ac5056435b7ecd4
Author: Vyacheslav Baranov <sl...@gmail.com>
Date: 2015-04-29T14:46:35Z
[SPARK-6913] Fixed "No suitable driver found" when using using JDBC driver added with SparkContext.addJar
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed "java.sql.SQLException...
Posted by SlavikBaranov <gi...@git.apache.org>.
Github user SlavikBaranov commented on the pull request:
https://github.com/apache/spark/pull/5782#issuecomment-97683350
Thanks for comments, fixed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed "java.sql.SQLException...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5782#issuecomment-97635442
[Test build #31370 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31370/consoleFull) for PR 5782 at commit [`b2a727c`](https://github.com/apache/spark/commit/b2a727c7ed4412c9b9d42b75c5a52369bd9ac43e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed "java.sql.SQLException...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5782#discussion_r29399579
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/jdbc/jdbc.scala ---
@@ -179,4 +184,63 @@ package object jdbc {
}
}
+
+ private [sql] case class DriverWrapper(wrapped: Driver) extends Driver {
--- End diff --
this doesn't need to be a case class, does it? Looks like you can just slightly change the pattern matching down below.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed "java.sql.SQLException...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5782#discussion_r29399631
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/jdbc/jdbc.scala ---
@@ -179,4 +184,63 @@ package object jdbc {
}
}
+
+ private [sql] case class DriverWrapper(wrapped: Driver) extends Driver {
+ override def acceptsURL(url: String): Boolean = wrapped.acceptsURL(url)
+
+ override def jdbcCompliant(): Boolean = wrapped.jdbcCompliant()
+
+ override def getPropertyInfo(url: String, info: Properties): Array[DriverPropertyInfo] = {
+ wrapped.getPropertyInfo(url, info)
+ }
+
+ override def getMinorVersion: Int = wrapped.getMinorVersion
+
+ override def getParentLogger: java.util.logging.Logger = wrapped.getParentLogger
+
+ override def connect(url: String, info: Properties): Connection = wrapped.connect(url, info)
+
+ override def getMajorVersion: Int = wrapped.getMajorVersion
+ }
+
+ /**
+ * java.sql.DriverManager is always loaded by bootstrap classloader,
+ * so it can't load JDBC drivers accessible by Spark ClassLoader.
+ *
+ * To solve the problem, drivers from user-supplied jars are wrapped
+ * into thin wrapper.
+ */
+ private [sql] object DriverRegistry extends Logging {
+
+ val wrapperMap: mutable.Map[String, DriverWrapper] = mutable.Map.empty
--- End diff --
private
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed "java.sql.SQLException...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/5782#issuecomment-97684199
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed "java.sql.SQLException...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5782#issuecomment-97635277
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed "java.sql.SQLException...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5782#issuecomment-97715156
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed "java.sql.SQLException...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5782#discussion_r29399610
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/jdbc/jdbc.scala ---
@@ -179,4 +184,63 @@ package object jdbc {
}
}
+
+ private [sql] case class DriverWrapper(wrapped: Driver) extends Driver {
+ override def acceptsURL(url: String): Boolean = wrapped.acceptsURL(url)
+
+ override def jdbcCompliant(): Boolean = wrapped.jdbcCompliant()
+
+ override def getPropertyInfo(url: String, info: Properties): Array[DriverPropertyInfo] = {
+ wrapped.getPropertyInfo(url, info)
+ }
+
+ override def getMinorVersion: Int = wrapped.getMinorVersion
+
+ override def getParentLogger: java.util.logging.Logger = wrapped.getParentLogger
+
+ override def connect(url: String, info: Properties): Connection = wrapped.connect(url, info)
+
+ override def getMajorVersion: Int = wrapped.getMajorVersion
+ }
+
+ /**
+ * java.sql.DriverManager is always loaded by bootstrap classloader,
+ * so it can't load JDBC drivers accessible by Spark ClassLoader.
+ *
+ * To solve the problem, drivers from user-supplied jars are wrapped
+ * into thin wrapper.
+ */
+ private [sql] object DriverRegistry extends Logging {
+
+ val wrapperMap: mutable.Map[String, DriverWrapper] = mutable.Map.empty
+
+ val lock = new ReentrantLock()
+
+ def register(className: String): Unit = {
+ val cls = Utils.getContextOrSparkClassLoader.loadClass(className)
+ if (cls.getClassLoader == null) {
+ logTrace(s"$className has been loaded with bootstrap ClassLoader, wrapper is not required")
+ } else if (wrapperMap.get(className).isDefined) {
+ logTrace(s"Wrapper for $className already exists")
+ } else {
+ lock.lock()
--- End diff --
given this is unlikely to be a performance bottleneck, why not just use a synchronized block?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed "java.sql.SQLException...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/5782#issuecomment-97635113
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed "java.sql.SQLException...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5782#issuecomment-97684830
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed "java.sql.SQLException...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5782#issuecomment-97685088
[Test build #31394 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31394/consoleFull) for PR 5782 at commit [`510c43f`](https://github.com/apache/spark/commit/510c43f691f4b3901ca877585e25b28121a0cf36).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed "java.sql.SQLException...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5782#issuecomment-97654329
[Test build #31370 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31370/consoleFull) for PR 5782 at commit [`b2a727c`](https://github.com/apache/spark/commit/b2a727c7ed4412c9b9d42b75c5a52369bd9ac43e).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed "java.sql.SQLException...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5782#issuecomment-97654336
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed "java.sql.SQLException...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5782#issuecomment-97715160
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31394/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed "java.sql.SQLException...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5782#issuecomment-97635289
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed "java.sql.SQLException...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5782#issuecomment-97457751
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed "java.sql.SQLException...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5782#issuecomment-97715129
[Test build #31394 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31394/consoleFull) for PR 5782 at commit [`510c43f`](https://github.com/apache/spark/commit/510c43f691f4b3901ca877585e25b28121a0cf36).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch **adds the following new dependencies:**
* `jaxb-api-2.2.7.jar`
* `jaxb-core-2.2.7.jar`
* `jaxb-impl-2.2.7.jar`
* `pmml-agent-1.1.15.jar`
* `pmml-model-1.1.15.jar`
* `pmml-schema-1.1.15.jar`
* This patch **removes the following dependencies:**
* `activation-1.1.jar`
* `jaxb-api-2.2.2.jar`
* `jaxb-impl-2.2.3-1.jar`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed "java.sql.SQLException...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5782#issuecomment-97654337
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31370/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6913][SQL] Fixed "java.sql.SQLException...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5782#issuecomment-97684816
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #5782: [SPARK-6913][SQL] Fixed "java.sql.SQLException: No suitab...
Posted by yubao2 <gi...@git.apache.org>.
Github user yubao2 commented on the issue:
https://github.com/apache/spark/pull/5782
你好，我是一个新手，遇到这样的问题不知道何从下手，希望可以得到有效的帮助，我是在使用sparkSQL将处理过的数据保存至mysql中，打包上传运行时出现的“java.sql.SQLException: No suitable driver”，spark版本是2.3.2，希望可以得到一个有效的解决办法，谢谢
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org