You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by davies <gi...@git.apache.org> on 2015/09/24 23:26:39 UTC
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
GitHub user davies opened a pull request:
https://github.com/apache/spark/pull/8909
[SPARK-10810] [WIP] [SQL] Improve session management in SQL
This PR improve the sessions management by replacing the thread-local based to one SQLContext per session approach, introduce separated temporary tables and UDFs/UDAFs for each session.
A new session of SQLContext could be created by:
1) create an new SQLContext
2) call newSession() on existing SQLContext
For HiveContext, in order to reduce the cost for each session, the classloader and Hive client are shared across multiple sessions (created by newSession).
CacheManager is also shared by multiple sessions, so cache a table multiple times in different sessions will not cause multiple copies of in-memory cache.
Added jars are still shared by all the sessions, because SparkContext does not support sessions.
cc @marmbrus @yhuai @rxin
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/davies/spark sessions
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/8909.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #8909
----
commit af8df73556b466fa68dd0689469cb465dd761a1e
Author: Davies Liu <da...@databricks.com>
Date: 2015-09-24T19:08:54Z
sessions for SQLContext
commit bc9f06447482dac5cfc61fd9d8c21bacb2300431
Author: Davies Liu <da...@databricks.com>
Date: 2015-09-24T21:18:00Z
improve executionHive
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146399329
[Test build #1856 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1856/console) for PR 8909 at commit [`3aeae47`](https://github.com/apache/spark/commit/3aeae474fdada66b38b8ef81aa1523880f1b8a8a).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` /** Run a function within Hive state (SessionState, HiveConf, Hive client and class loader) */`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41421146
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala ---
@@ -89,6 +80,7 @@ private[sql] class CacheManager(sqlContext: SQLContext) extends Logging {
* the in-memory columnar representation of the underlying table is expensive.
*/
private[sql] def cacheQuery(
+ sqlContext: SQLContext,
--- End diff --
You can get the `sqlContext` from `query` right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41541665
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -1196,49 +1174,101 @@ class SQLContext(@transient val sparkContext: SparkContext)
// Register a succesfully instantiatd context to the singleton. This should be at the end of
// the class definition so that the singleton is updated only if there is no exception in the
// construction of the instance.
- SQLContext.setLastInstantiatedContext(self)
+ sparkContext.addSparkListener(new SparkListener{
+ override def onApplicationEnd(applicationEnd: SparkListenerApplicationEnd) {
--- End diff --
@andrewor14 The `SparkListener` does not have `: Unit`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-143056917
[Test build #42982 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42982/console) for PR 8909 at commit [`bc9f064`](https://github.com/apache/spark/commit/bc9f06447482dac5cfc61fd9d8c21bacb2300431).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SQL] Improve session management...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144903102
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SQL] Improve session management...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144896062
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43165/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144577494
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by Neuw84 <gi...@git.apache.org>.
Github user Neuw84 commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-169046278
Since upgrading to Spark 1.6 I can´t query temp tables created from SparkStreaming using Apache Toad.
The relevant Spark Streaming part.
window.foreachRDD((JavaRDD<Row> rdd, Time time) -> {
if (!rdd.isEmpty()) {
HiveContext sqlContext1 = JavaSQLContextSingleton.getInstance(rdd.context());
if (sqlContext1.isCached("issues_recent")) {
sqlContext1.uncacheTable("issues_recent");
}
DataFrame recentDataFrame = sqlContext1.createDataFrame(rdd, schema).coalesce(20);
recentDataFrame.registerTempTable("issues_recent");
recentDataFrame.cache();
}
The SQLContext creation part
class JavaSQLContextSingleton {
static private transient HiveContext instance = null;
static public HiveContext getInstance(SparkContext sparkContext) {
if (instance == null) {
instance = new HiveContext(sparkContext);
}
return instance;
}
}
Without using spark streaming doesn´t help with the problem (loading parque file a registering a temporary table).
I think that this "issue" may be related to this PR (maybe I am wrong and this is not the correct place to put this).
Any hints?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SQL] Improve session management...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144896061
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146683288
[Test build #43428 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43428/consoleFull) for PR 8909 at commit [`ab5e46c`](https://github.com/apache/spark/commit/ab5e46cc79c5d3fe4c1b0662ac58ed07f3b94f34).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146668852
[Test build #43407 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43407/console) for PR 8909 at commit [`19b91df`](https://github.com/apache/spark/commit/19b91dfc1c6ce16409411dbb54925dcebc39e608).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` /** Run a function within Hive state (SessionState, HiveConf, Hive client and class loader) */`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146353081
@yhuai `ADD JAR` with SERDE is already covered well by unit tests (including JDBC client and Cli and HiveContext). `--jar` is tested manually, work well. Which jar could be used to test UDF?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SQL] Improve session management...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144862831
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43160/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SQL] Improve session management...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144831822
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41428732
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala ---
@@ -410,7 +436,11 @@ class HiveContext(sc: SparkContext) extends SQLContext(sc) with Logging {
// Note that HiveUDFs will be overridden by functions registered in this context.
@transient
override protected[sql] lazy val functionRegistry: FunctionRegistry =
- new HiveFunctionRegistry(FunctionRegistry.builtin)
+ new HiveFunctionRegistry(FunctionRegistry.builtin.copy())
+
+ functionRegistry.registerFunction(
+ "current_database",
+ (expressions: Seq[Expression]) => new CurrentDatabase(this))
--- End diff --
Add a comment to explain why we need to do it at here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146006460
[Test build #43287 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43287/console) for PR 8909 at commit [`ae0e876`](https://github.com/apache/spark/commit/ae0e87677a6c27532f071c4f064d42805029b53d).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` /** Run a function within Hive state (SessionState, HiveConf, Hive client and class loader) */`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146370739
[Test build #43357 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43357/console) for PR 8909 at commit [`3aeae47`](https://github.com/apache/spark/commit/3aeae474fdada66b38b8ef81aa1523880f1b8a8a).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` /** Run a function within Hive state (SessionState, HiveConf, Hive client and class loader) */`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41462111
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -1196,49 +1174,101 @@ class SQLContext(@transient val sparkContext: SparkContext)
// Register a succesfully instantiatd context to the singleton. This should be at the end of
// the class definition so that the singleton is updated only if there is no exception in the
// construction of the instance.
- SQLContext.setLastInstantiatedContext(self)
+ sparkContext.addSparkListener(new SparkListener{
+ override def onApplicationEnd(applicationEnd: SparkListenerApplicationEnd) {
--- End diff --
`Unit`, also space before `{` in previous line
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41462336
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -1196,49 +1174,101 @@ class SQLContext(@transient val sparkContext: SparkContext)
// Register a succesfully instantiatd context to the singleton. This should be at the end of
// the class definition so that the singleton is updated only if there is no exception in the
// construction of the instance.
- SQLContext.setLastInstantiatedContext(self)
+ sparkContext.addSparkListener(new SparkListener{
+ override def onApplicationEnd(applicationEnd: SparkListenerApplicationEnd) {
+ SQLContext.clearTheInstantiatedContext(self)
+ }
+ })
+
+ SQLContext.setInstantiatedContext(self)
}
/**
* This SQLContext object contains utility functions to create a singleton SQLContext instance,
- * or to get the last created SQLContext instance.
+ * or to get the created SQLContext instance.
*/
object SQLContext {
private val INSTANTIATION_LOCK = new Object()
/**
- * Reference to the last created SQLContext.
+ * The active SQLContext for the current thread.
*/
- @transient private val lastInstantiatedContext = new AtomicReference[SQLContext]()
+ private val activeContext: InheritableThreadLocal[SQLContext] =
+ new InheritableThreadLocal[SQLContext]
+
+ /**
+ * Reference to the created SQLContext.
+ */
+ @transient private val instantiatedContext = new AtomicReference[SQLContext]()
/**
* Get the singleton SQLContext if it exists or create a new one using the given SparkContext.
+ *
* This function can be used to create a singleton SQLContext object that can be shared across
* the JVM.
+ *
+ * If there is an active SQLContext for current thread, it will returned instead of the global
--- End diff --
it will returned -> it will be returned
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41424269
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -64,18 +63,28 @@ import org.apache.spark.util.Utils
*
* @since 1.0.0
*/
-class SQLContext(@transient val sparkContext: SparkContext)
- extends org.apache.spark.Logging
- with Serializable {
+class SQLContext private[sql](
+ @transient val sparkContext: SparkContext,
+ @transient protected[sql] val cacheManager: CacheManager)
+ extends org.apache.spark.Logging with Serializable {
self =>
+ def this(sparkContext: SparkContext) = this(sparkContext, new CacheManager)
def this(sparkContext: JavaSparkContext) = this(sparkContext.sc)
/**
+ * Returns a SQLContext as new session, with separated SQL configurations, temporary tables,
+ * registered functions, but share the same SparkContext and CacheManager.
+ */
+ def newSession(): SQLContext = {
+ new SQLContext(sparkContext, cacheManager)
+ }
+
+ /**
* @return Spark SQL configuration
*/
- protected[sql] def conf = currentSession().conf
+ protected[sql] lazy val conf = new SQLConf
--- End diff --
Can we avoid of using `lazy val`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144578203
[Test build #43141 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43141/consoleFull) for PR 8909 at commit [`bcaddb3`](https://github.com/apache/spark/commit/bcaddb31cbeb10d8eb06a5397b5059417818b942).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41422438
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala ---
@@ -148,53 +150,88 @@ private[hive] class IsolatedClientLoader(
name.replaceAll("\\.", "/") + ".class"
/** The classloader that is used to load an isolated version of Hive. */
- protected val classLoader: ClassLoader = new URLClassLoader(allJars, rootClassLoader) {
- override def loadClass(name: String, resolve: Boolean): Class[_] = {
- val loaded = findLoadedClass(name)
- if (loaded == null) doLoadClass(name, resolve) else loaded
- }
+ private[hive] var classLoader: ClassLoader = if (isolationOn) {
+ new URLClassLoader(allJars, rootClassLoader) {
+ val cache = new java.util.concurrent.ConcurrentHashMap[String, Class[_]]
+
+ override def loadClass(name: String, resolve: Boolean): Class[_] = {
+ var clazz = findLoadedClass(name)
+ if (clazz == null) {
+ clazz = cache.get(name)
+ if (clazz == null) {
+ // always assume resolve is false at this point - we'll resolve later
+ clazz = doLoadClass(name, resolve = false)
+ val clazz2 = cache.putIfAbsent(name, clazz)
+ // check if someone else beat us to updating the cache entry
+ if (clazz2 != null) {
+ // if so then we should use the cached entry to be consistent
+ clazz = clazz2
+ }
+ }
+ if (resolve) {
+ resolveClass(clazz)
+ }
+ }
+ clazz
+ }
- def doLoadClass(name: String, resolve: Boolean): Class[_] = {
- val classFileName = name.replaceAll("\\.", "/") + ".class"
- if (isBarrierClass(name) && isolationOn) {
- // For barrier classes, we construct a new copy of the class.
- val bytes = IOUtils.toByteArray(baseClassLoader.getResourceAsStream(classFileName))
- logDebug(s"custom defining: $name - ${util.Arrays.hashCode(bytes)}")
- defineClass(name, bytes, 0, bytes.length)
- } else if (!isSharedClass(name)) {
- logDebug(s"hive class: $name - ${getResource(classToPath(name))}")
- super.loadClass(name, resolve)
- } else {
- // For shared classes, we delegate to baseClassLoader.
- logDebug(s"shared class: $name")
- baseClassLoader.loadClass(name)
+ def doLoadClass(name: String, resolve: Boolean): Class[_] = {
+ val classFileName = name.replaceAll("\\.", "/") + ".class"
+ if (isBarrierClass(name)) {
+ // For barrier classes, we construct a new copy of the class.
+ val bytes = IOUtils.toByteArray(baseClassLoader.getResourceAsStream(classFileName))
+ logDebug(s"custom defining: $name - ${util.Arrays.hashCode(bytes)}")
+ defineClass(name, bytes, 0, bytes.length)
+ } else if (!isSharedClass(name)) {
+ logDebug(s"hive class: $name - ${getResource(classToPath(name))}")
+ super.loadClass(name, resolve)
+ } else {
+ // For shared classes, we delegate to baseClassLoader.
+ logDebug(s"shared class: $name")
+ baseClassLoader.loadClass(name)
+ }
}
}
+ } else {
+ baseClassLoader
}
- // Pre-reflective instantiation setup.
- logDebug("Initializing the logger to avoid disaster...")
- Thread.currentThread.setContextClassLoader(classLoader)
+ private[hive] def addJar(path: String): Unit = synchronized {
+ val jarURL = new java.io.File(path).toURI.toURL
+ classLoader = new java.net.URLClassLoader(Array(jarURL), classLoader)
+ }
/** The isolated client interface to Hive. */
- val client: ClientInterface = try {
- classLoader
- .loadClass(classOf[ClientWrapper].getName)
- .getConstructors.head
- .newInstance(version, config, classLoader)
- .asInstanceOf[ClientInterface]
- } catch {
- case e: InvocationTargetException =>
- if (e.getCause().isInstanceOf[NoClassDefFoundError]) {
- val cnf = e.getCause().asInstanceOf[NoClassDefFoundError]
- throw new ClassNotFoundException(
- s"$cnf when creating Hive client using classpath: ${execJars.mkString(", ")}\n" +
- "Please make sure that jars for your version of hive and hadoop are included in the " +
- s"paths passed to ${HiveContext.HIVE_METASTORE_JARS}.")
- } else {
- throw e
- }
- } finally {
- Thread.currentThread.setContextClassLoader(baseClassLoader)
+ private[hive] def createClient(): ClientInterface = {
+ if (!isolationOn) {
+ return new ClientWrapper(version, config, baseClassLoader, this)
+ }
+ // Pre-reflective instantiation setup.
+ logDebug("Initializing the logger to avoid disaster...")
+ val origLoader = Thread.currentThread().getContextClassLoader
+ Thread.currentThread.setContextClassLoader(classLoader)
+
+ try {
+ classLoader
+ .loadClass(classOf[ClientWrapper].getName)
+ .getConstructors.head
+ .newInstance(version, config, classLoader, this)
+ .asInstanceOf[ClientInterface]
+ } catch {
+ case e: InvocationTargetException =>
+ if (e.getCause().isInstanceOf[NoClassDefFoundError]) {
+ val cnf = e.getCause().asInstanceOf[NoClassDefFoundError]
+ throw new ClassNotFoundException(
+ s"$cnf when creating Hive client using classpath: ${execJars.mkString(", ")}\n" +
+ "Please make sure that jars for your version of hive and hadoop are included in the " +
+ s"paths passed to ${HiveContext.HIVE_METASTORE_JARS}.")
+ } else {
+ throw e
+ }
+ } finally {
+ Thread.currentThread.setContextClassLoader(origLoader)
+ }
}
+
+ private[hive] var cachedHive: Any = null
--- End diff --
Can you document what this is used for.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by deenar <gi...@git.apache.org>.
Github user deenar commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-175871622
@Neuw84, i presume you are connecting TOAD to Spark SQL via the thrift server. You might need to set spark.sql.hive.thriftServer.singleSession=true to preserve the pre 1.6 behaviour
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-143107956
[Test build #42997 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42997/console) for PR 8909 at commit [`9c01c27`](https://github.com/apache/spark/commit/9c01c27f9a8b96a86f2260a2aeb8f942771b054c).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SQL] Improve session management...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144863817
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144599640
[Test build #43141 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43141/console) for PR 8909 at commit [`bcaddb3`](https://github.com/apache/spark/commit/bcaddb31cbeb10d8eb06a5397b5059417818b942).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class HyperLogLogPlusPlus(child: Expression, relativeSD: Double = 0.05)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144577473
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SQL] Improve session management...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144800460
[Test build #43155 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43155/consoleFull) for PR 8909 at commit [`9fe30cf`](https://github.com/apache/spark/commit/9fe30cf6372fd5c78a6aae004e19bd038f1cc93f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146367482
[Test build #43356 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43356/consoleFull) for PR 8909 at commit [`e16737c`](https://github.com/apache/spark/commit/e16737c18f2662da1ddfd00feb9e5f433ec94184).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-143107995
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-143089392
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144568767
[Test build #43137 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43137/console) for PR 8909 at commit [`e942e93`](https://github.com/apache/spark/commit/e942e9333274527665d9a72d4a9ceb14ae711854).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class HyperLogLogPlusPlus(child: Expression, relativeSD: Double = 0.05)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-143081979
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SQL] Improve session management...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144816856
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43158/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146006629
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43287/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-143081980
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42991/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41450931
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala ---
@@ -1133,6 +1133,17 @@ class HiveQuerySuite extends HiveComparisonTest with BeforeAndAfter {
conf.clear()
}
+ test("current_database with mutiple sessions") {
+ sql("create database a")
+ sql("use a")
+ val s2 = newSession()
+ s2.sql("create database b")
+ s2.sql("use b")
+
+ assert(sql("select current_database()").first() === Row("a"))
+ assert(s2.sql("select current_database()").first() === Row("b"))
--- End diff --
Yeah, the thrift server tests are so expensive / flakey that I'd like to have another cheaper unit test for all of this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41455693
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -64,18 +63,28 @@ import org.apache.spark.util.Utils
*
* @since 1.0.0
*/
-class SQLContext(@transient val sparkContext: SparkContext)
- extends org.apache.spark.Logging
- with Serializable {
+class SQLContext private[sql](
+ @transient val sparkContext: SparkContext,
+ @transient protected[sql] val cacheManager: CacheManager)
+ extends org.apache.spark.Logging with Serializable {
self =>
+ def this(sparkContext: SparkContext) = this(sparkContext, new CacheManager)
def this(sparkContext: JavaSparkContext) = this(sparkContext.sc)
/**
+ * Returns a SQLContext as new session, with separated SQL configurations, temporary tables,
+ * registered functions, but share the same SparkContext and CacheManager.
+ */
+ def newSession(): SQLContext = {
+ new SQLContext(sparkContext, cacheManager)
+ }
+
+ /**
* @return Spark SQL configuration
*/
- protected[sql] def conf = currentSession().conf
+ protected[sql] lazy val conf = new SQLConf
--- End diff --
`conf` will be override in HiveContext, but the constructor of SQLContext need to access that, so it can't be `val`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146715308
[Test build #43428 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43428/console) for PR 8909 at commit [`ab5e46c`](https://github.com/apache/spark/commit/ab5e46cc79c5d3fe4c1b0662ac58ed07f3b94f34).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144599664
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #8909: [SPARK-10810] [SPARK-10902] [SQL] Improve session ...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r71047738
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala ---
@@ -598,6 +631,14 @@ class HiveContext(sc: SparkContext) extends SQLContext(sc) with Logging {
case _ => super.simpleString
}
}
+
+ protected[sql] override def addJar(path: String): Unit = {
+ // Add jar to Hive and classloader
+ executionHive.addJar(path)
+ metadataHive.addJar(path)
+ Thread.currentThread().setContextClassLoader(executionHive.clientLoader.classLoader)
--- End diff --
@davies @yhuai Do we need to restore the original classloader after calling `super.addJar()` or is it intentional that calling `HiveContext.addJar` will permanently mutate the calling thread's context classloader?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146370209
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41455637
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -1196,48 +1174,93 @@ class SQLContext(@transient val sparkContext: SparkContext)
// Register a succesfully instantiatd context to the singleton. This should be at the end of
// the class definition so that the singleton is updated only if there is no exception in the
// construction of the instance.
- SQLContext.setLastInstantiatedContext(self)
+ sparkContext.addSparkListener(new SparkListener{
+ override def onApplicationEnd(applicationEnd: SparkListenerApplicationEnd) {
+ SQLContext.clearTheInstantiatedContext(self)
+ }
+ })
+
+ SQLContext.setInstantiatedContext(self)
}
/**
* This SQLContext object contains utility functions to create a singleton SQLContext instance,
- * or to get the last created SQLContext instance.
+ * or to get the created SQLContext instance.
*/
object SQLContext {
private val INSTANTIATION_LOCK = new Object()
/**
- * Reference to the last created SQLContext.
+ * The active SQLContext for the current thread.
*/
- @transient private val lastInstantiatedContext = new AtomicReference[SQLContext]()
+ private val activeContext: InheritableThreadLocal[SQLContext] =
+ new InheritableThreadLocal[SQLContext]
+
+ /**
+ * Reference to the created SQLContext.
+ */
+ @transient private val instantiatedContext = new AtomicReference[SQLContext]()
/**
* Get the singleton SQLContext if it exists or create a new one using the given SparkContext.
+ *
* This function can be used to create a singleton SQLContext object that can be shared across
* the JVM.
+ *
+ * If there an active SQLContext for current thread, it will returned instead of the global one.
--- End diff --
Nit: "If there **is** an active ..."
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-143078227
[Test build #1814 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1814/consoleFull) for PR 8909 at commit [`0cf26a5`](https://github.com/apache/spark/commit/0cf26a57c844bd7379e07f518555418b0b3ecefb).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144798664
[Test build #43154 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43154/console) for PR 8909 at commit [`2e365ad`](https://github.com/apache/spark/commit/2e365ada232e42d88692c89634e5ed2ceb741beb).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class HyperLogLogPlusPlus(child: Expression, relativeSD: Double = 0.05)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-143073092
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by tedyu <gi...@git.apache.org>.
Github user tedyu commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r42816551
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -1196,46 +1174,90 @@ class SQLContext(@transient val sparkContext: SparkContext)
// Register a succesfully instantiatd context to the singleton. This should be at the end of
// the class definition so that the singleton is updated only if there is no exception in the
// construction of the instance.
- SQLContext.setLastInstantiatedContext(self)
+ sparkContext.addSparkListener(new SparkListener {
+ override def onApplicationEnd(applicationEnd: SparkListenerApplicationEnd): Unit = {
+ SQLContext.clearInstantiatedContext(self)
+ }
+ })
+
+ SQLContext.setInstantiatedContext(self)
}
/**
* This SQLContext object contains utility functions to create a singleton SQLContext instance,
- * or to get the last created SQLContext instance.
+ * or to get the created SQLContext instance.
+ *
+ * It also provides utility functions to support preference for threads in multiple sessions
+ * scenario, setActive could set a SQLContext for current thread, which will be returned by
+ * getOrCreate instead of the global one.
*/
object SQLContext {
- private val INSTANTIATION_LOCK = new Object()
+ /**
+ * The active SQLContext for the current thread.
+ */
+ private val activeContext: InheritableThreadLocal[SQLContext] =
--- End diff --
Should this be annotated @volatile ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-143071378
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SQL] Improve session management...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144867433
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43162/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144797968
[Test build #43154 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43154/consoleFull) for PR 8909 at commit [`2e365ad`](https://github.com/apache/spark/commit/2e365ada232e42d88692c89634e5ed2ceb741beb).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41462407
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -1196,49 +1174,101 @@ class SQLContext(@transient val sparkContext: SparkContext)
// Register a succesfully instantiatd context to the singleton. This should be at the end of
// the class definition so that the singleton is updated only if there is no exception in the
// construction of the instance.
- SQLContext.setLastInstantiatedContext(self)
+ sparkContext.addSparkListener(new SparkListener{
+ override def onApplicationEnd(applicationEnd: SparkListenerApplicationEnd) {
+ SQLContext.clearTheInstantiatedContext(self)
+ }
+ })
+
+ SQLContext.setInstantiatedContext(self)
}
/**
* This SQLContext object contains utility functions to create a singleton SQLContext instance,
- * or to get the last created SQLContext instance.
+ * or to get the created SQLContext instance.
*/
object SQLContext {
private val INSTANTIATION_LOCK = new Object()
/**
- * Reference to the last created SQLContext.
+ * The active SQLContext for the current thread.
*/
- @transient private val lastInstantiatedContext = new AtomicReference[SQLContext]()
+ private val activeContext: InheritableThreadLocal[SQLContext] =
--- End diff --
it's not clear how this is different from `instantiatedContext` from the name. Maybe this should be `threadLocalContext` or `threadActiveContext` or something
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SQL] Improve session management...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144928604
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SQL] Improve session management...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144831826
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43155/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by chutium <gi...@git.apache.org>.
Github user chutium commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-198071715
anyone tried use --hivevar or --hiveconf with beeline after this change? variables from JDBC client is no more available in the hive session.
refer to https://issues.apache.org/jira/browse/SPARK-13983
is there any test case for --hivevar and --hiveconf in test suites?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-143107996
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42997/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144525589
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-143055408
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r42817021
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -1196,46 +1174,90 @@ class SQLContext(@transient val sparkContext: SparkContext)
// Register a succesfully instantiatd context to the singleton. This should be at the end of
// the class definition so that the singleton is updated only if there is no exception in the
// construction of the instance.
- SQLContext.setLastInstantiatedContext(self)
+ sparkContext.addSparkListener(new SparkListener {
+ override def onApplicationEnd(applicationEnd: SparkListenerApplicationEnd): Unit = {
+ SQLContext.clearInstantiatedContext(self)
+ }
+ })
+
+ SQLContext.setInstantiatedContext(self)
}
/**
* This SQLContext object contains utility functions to create a singleton SQLContext instance,
- * or to get the last created SQLContext instance.
+ * or to get the created SQLContext instance.
+ *
+ * It also provides utility functions to support preference for threads in multiple sessions
+ * scenario, setActive could set a SQLContext for current thread, which will be returned by
+ * getOrCreate instead of the global one.
*/
object SQLContext {
- private val INSTANTIATION_LOCK = new Object()
+ /**
+ * The active SQLContext for the current thread.
+ */
+ private val activeContext: InheritableThreadLocal[SQLContext] =
+ new InheritableThreadLocal[SQLContext]
/**
- * Reference to the last created SQLContext.
+ * Reference to the created SQLContext.
*/
- @transient private val lastInstantiatedContext = new AtomicReference[SQLContext]()
+ @transient private val instantiatedContext = new AtomicReference[SQLContext]()
--- End diff --
Yeah agreed. In fact @volatile should never be needed on a val.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41449487
--- Diff: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala ---
@@ -289,16 +293,15 @@ class HiveThriftBinaryServerSuite extends HiveThriftJdbcTest {
rs2.close()
},
- // accessing the cached data in another session
+ // try to access the cached data in another session
{ statement =>
- val rs1 = statement.executeQuery("SELECT key FROM test_table ORDER BY KEY DESC")
- val buf1 = new collection.mutable.ArrayBuffer[Int]()
- while (rs1.next()) {
- buf1 += rs1.getInt(1)
+ // Cached temporary table can't be accessed by other sessions
+ intercept[SQLException] {
+ statement.executeQuery("SELECT key FROM test_table ORDER BY KEY DESC")
}
- rs1.close()
+ // TODO need to figure out how to determine if the data loaded from cache
--- End diff --
This is JDBC client, so we can't access the internal of catalyst. Maybe we could use explain().
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146725227
Non-hive parts LGTM!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146634524
[Test build #43407 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43407/consoleFull) for PR 8909 at commit [`19b91df`](https://github.com/apache/spark/commit/19b91dfc1c6ce16409411dbb54925dcebc39e608).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41420887
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -1208,36 +1204,72 @@ object SQLContext {
private val INSTANTIATION_LOCK = new Object()
/**
- * Reference to the last created SQLContext.
+ * The active SQLContext for threads.
--- End diff --
The active SQLContext for _the current thread_.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144798673
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43154/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SQL] Improve session management...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144863796
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146699044
@marmbrus @yhuai @andrewor14 I should have addressed all of your comments (except lack of more docs), could you take another pass? The failed test is not related to this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SQL] Improve session management...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144799890
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-143073086
[Test build #42988 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42988/console) for PR 8909 at commit [`0cf26a5`](https://github.com/apache/spark/commit/0cf26a57c844bd7379e07f518555418b0b3ecefb).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SQL] Improve session management...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144812883
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146370861
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43357/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41449262
--- Diff: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala ---
@@ -159,7 +159,7 @@ class CliSuite extends SparkFunSuite with BeforeAndAfter with Logging {
s"LOAD DATA LOCAL INPATH '$dataFilePath' OVERWRITE INTO TABLE hive_test;"
-> "OK",
"CACHE TABLE hive_test;"
- -> "Time taken: ",
+ -> "",
--- End diff --
The output does not always have `Time taken`, test will fail.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146281028
@davies, this is great. Much cleaner than the old implementation!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-143056919
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144599666
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43141/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-143090113
[Test build #42997 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42997/consoleFull) for PR 8909 at commit [`9c01c27`](https://github.com/apache/spark/commit/9c01c27f9a8b96a86f2260a2aeb8f942771b054c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146715402
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-143055366
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41462815
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -1196,49 +1174,101 @@ class SQLContext(@transient val sparkContext: SparkContext)
// Register a succesfully instantiatd context to the singleton. This should be at the end of
// the class definition so that the singleton is updated only if there is no exception in the
// construction of the instance.
- SQLContext.setLastInstantiatedContext(self)
+ sparkContext.addSparkListener(new SparkListener{
+ override def onApplicationEnd(applicationEnd: SparkListenerApplicationEnd) {
+ SQLContext.clearTheInstantiatedContext(self)
+ }
+ })
+
+ SQLContext.setInstantiatedContext(self)
}
/**
* This SQLContext object contains utility functions to create a singleton SQLContext instance,
- * or to get the last created SQLContext instance.
+ * or to get the created SQLContext instance.
*/
object SQLContext {
private val INSTANTIATION_LOCK = new Object()
/**
- * Reference to the last created SQLContext.
+ * The active SQLContext for the current thread.
*/
- @transient private val lastInstantiatedContext = new AtomicReference[SQLContext]()
+ private val activeContext: InheritableThreadLocal[SQLContext] =
+ new InheritableThreadLocal[SQLContext]
+
+ /**
+ * Reference to the created SQLContext.
+ */
+ @transient private val instantiatedContext = new AtomicReference[SQLContext]()
--- End diff --
if we absolutely have to keep both, then we should expand on the comments to explain why a thread local context is needed in addition to one that was initially instantiated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146668949
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43407/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SQL] Improve session management...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144862464
[Test build #43160 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43160/consoleFull) for PR 8909 at commit [`d10a97a`](https://github.com/apache/spark/commit/d10a97a23608e2f27bbeb6760559972b26d64c56).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144797039
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-143056921
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42982/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146633072
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41420084
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala ---
@@ -51,22 +51,42 @@ class SimpleFunctionRegistry extends FunctionRegistry {
private val functionBuilders =
StringKeyHashMap[(ExpressionInfo, FunctionBuilder)](caseSensitive = false)
- override def registerFunction(name: String, info: ExpressionInfo, builder: FunctionBuilder)
- : Unit = {
- functionBuilders.put(name, (info, builder))
+ override def registerFunction(
+ name: String,
+ info: ExpressionInfo,
+ builder: FunctionBuilder): Unit = {
+ synchronized {
--- End diff --
Style nit: in most places we put `synchronized` inline with the return type.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-145958687
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41450970
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala ---
@@ -148,53 +150,88 @@ private[hive] class IsolatedClientLoader(
name.replaceAll("\\.", "/") + ".class"
/** The classloader that is used to load an isolated version of Hive. */
- protected val classLoader: ClassLoader = new URLClassLoader(allJars, rootClassLoader) {
- override def loadClass(name: String, resolve: Boolean): Class[_] = {
- val loaded = findLoadedClass(name)
- if (loaded == null) doLoadClass(name, resolve) else loaded
- }
+ private[hive] var classLoader: ClassLoader = if (isolationOn) {
+ new URLClassLoader(allJars, rootClassLoader) {
+ val cache = new java.util.concurrent.ConcurrentHashMap[String, Class[_]]
--- End diff --
will remove the cache in this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-143081918
[Test build #42991 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42991/console) for PR 8909 at commit [`3723768`](https://github.com/apache/spark/commit/372376862ccbf1877dbbb2a26761c76d4b3afdd5).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144568840
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43137/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-169077537
@Neuw84 Since 1.6, each SQLContext will have separated session, you should share the same SQLContext in this case (using SQLContext.getOrCreate()).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41449894
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala ---
@@ -136,7 +137,8 @@ private[hive] class IsolatedClientLoader(
(name.startsWith("com.google") && !name.startsWith("com.google.cloud")) ||
name.startsWith("java.lang.") ||
name.startsWith("java.net") ||
- sharedPrefixes.exists(name.startsWith)
+ sharedPrefixes.exists(name.startsWith) ||
+ name.startsWith("[")
--- End diff --
Just saw that many array types are loaded multiple times, it's not that necessary for this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41423130
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala ---
@@ -1133,6 +1133,17 @@ class HiveQuerySuite extends HiveComparisonTest with BeforeAndAfter {
conf.clear()
}
+ test("current_database with mutiple sessions") {
+ sql("create database a")
+ sql("use a")
+ val s2 = newSession()
+ s2.sql("create database b")
+ s2.sql("use b")
+
+ assert(sql("select current_database()").first() === Row("a"))
+ assert(s2.sql("select current_database()").first() === Row("b"))
--- End diff --
Can we also add a test that tries querying from tables that only exist in one database or another to make sure that the catalogs resolution of "current database" is working.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146281503
@navis, that does sound like something we should fix, but unless we are breaking that here (and I don't think we are). I'd rather do this in a followup PR rather than block this large easily conflicted change. Please correct me if I'm wrong though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146682934
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-145958655
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146294817
This is super nice! It will be great if we can manually test the following cases to make sure Hive UDFs and SerDes added through user jars are good:
* a spark-shell with jars loaded through `--jars` option.
* a spark-shell with jars loaded through `add jar` command.
* a beeline client with jars loaded through `add jar` command.
* a cli client with jars loaded through `add jar` command.
* a cli client with jars loaded through `--jars` option.
Regarding commands to test, we can try commands that involve UDFs (register a UDF and use it) and SerDes (define a table, write data to it and query it).
Also, can we put a diagram in the comment (maybe in a follow-up pr) to explain relationship among HiveContext, ClientWrapper, and IsolatedLoader?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-145959378
[Test build #43287 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43287/consoleFull) for PR 8909 at commit [`ae0e876`](https://github.com/apache/spark/commit/ae0e87677a6c27532f071c4f064d42805029b53d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41450098
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala ---
@@ -1133,6 +1133,17 @@ class HiveQuerySuite extends HiveComparisonTest with BeforeAndAfter {
conf.clear()
}
+ test("current_database with mutiple sessions") {
+ sql("create database a")
+ sql("use a")
+ val s2 = newSession()
+ s2.sql("create database b")
+ s2.sql("use b")
+
+ assert(sql("select current_database()").first() === Row("a"))
+ assert(s2.sql("select current_database()").first() === Row("b"))
--- End diff --
That is already covered by HiveThriftServerSuite, could also addd that here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by tedyu <gi...@git.apache.org>.
Github user tedyu commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r42816849
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -1196,46 +1174,90 @@ class SQLContext(@transient val sparkContext: SparkContext)
// Register a succesfully instantiatd context to the singleton. This should be at the end of
// the class definition so that the singleton is updated only if there is no exception in the
// construction of the instance.
- SQLContext.setLastInstantiatedContext(self)
+ sparkContext.addSparkListener(new SparkListener {
+ override def onApplicationEnd(applicationEnd: SparkListenerApplicationEnd): Unit = {
+ SQLContext.clearInstantiatedContext(self)
+ }
+ })
+
+ SQLContext.setInstantiatedContext(self)
}
/**
* This SQLContext object contains utility functions to create a singleton SQLContext instance,
- * or to get the last created SQLContext instance.
+ * or to get the created SQLContext instance.
+ *
+ * It also provides utility functions to support preference for threads in multiple sessions
+ * scenario, setActive could set a SQLContext for current thread, which will be returned by
+ * getOrCreate instead of the global one.
*/
object SQLContext {
- private val INSTANTIATION_LOCK = new Object()
+ /**
+ * The active SQLContext for the current thread.
+ */
+ private val activeContext: InheritableThreadLocal[SQLContext] =
+ new InheritableThreadLocal[SQLContext]
/**
- * Reference to the last created SQLContext.
+ * Reference to the created SQLContext.
*/
- @transient private val lastInstantiatedContext = new AtomicReference[SQLContext]()
+ @transient private val instantiatedContext = new AtomicReference[SQLContext]()
--- End diff --
Should this be annotated @volatile ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146354744
@davies Thanks for checking! For UDF, we can use `hive-contrib-0.13.1.jar` (https://github.com/apache/spark/blob/master/sql/hive/src/test/resources/hive-contrib-0.13.1.jar) which contains a udaf called `org.apache.hadoop.hive.contrib.udaf.example.UDAFExampleMax`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146725869
Merged into master, new comments will be addressed in follow-up PR, thanks you all!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41429939
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala ---
@@ -264,12 +262,16 @@ private[hive] object HiveQl extends Logging {
* Returns the HiveConf
*/
private[this] def hiveConf: HiveConf = {
- val ss = SessionState.get() // SessionState is lazy initialization, it can be null here
+ var ss = SessionState.get()
+ // SessionState is lazy initialization, it can be null here
if (ss == null) {
- new HiveConf()
- } else {
- ss.getConf
+ val original = Thread.currentThread().getContextClassLoader
+ val conf = new HiveConf(classOf[SessionState])
+ conf.setClassLoader(original)
+ ss = new SessionState(conf)
+ SessionState.start(ss)
--- End diff --
`SessionState.start` will change the context classloader to the classloader associated with `conf`. Is it the expected behavior?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41428482
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala ---
@@ -148,53 +150,88 @@ private[hive] class IsolatedClientLoader(
name.replaceAll("\\.", "/") + ".class"
/** The classloader that is used to load an isolated version of Hive. */
- protected val classLoader: ClassLoader = new URLClassLoader(allJars, rootClassLoader) {
- override def loadClass(name: String, resolve: Boolean): Class[_] = {
- val loaded = findLoadedClass(name)
- if (loaded == null) doLoadClass(name, resolve) else loaded
- }
+ private[hive] var classLoader: ClassLoader = if (isolationOn) {
+ new URLClassLoader(allJars, rootClassLoader) {
+ val cache = new java.util.concurrent.ConcurrentHashMap[String, Class[_]]
+
+ override def loadClass(name: String, resolve: Boolean): Class[_] = {
+ var clazz = findLoadedClass(name)
+ if (clazz == null) {
+ clazz = cache.get(name)
+ if (clazz == null) {
+ // always assume resolve is false at this point - we'll resolve later
+ clazz = doLoadClass(name, resolve = false)
+ val clazz2 = cache.putIfAbsent(name, clazz)
+ // check if someone else beat us to updating the cache entry
+ if (clazz2 != null) {
+ // if so then we should use the cached entry to be consistent
+ clazz = clazz2
+ }
+ }
+ if (resolve) {
+ resolveClass(clazz)
+ }
+ }
+ clazz
+ }
- def doLoadClass(name: String, resolve: Boolean): Class[_] = {
- val classFileName = name.replaceAll("\\.", "/") + ".class"
- if (isBarrierClass(name) && isolationOn) {
- // For barrier classes, we construct a new copy of the class.
- val bytes = IOUtils.toByteArray(baseClassLoader.getResourceAsStream(classFileName))
- logDebug(s"custom defining: $name - ${util.Arrays.hashCode(bytes)}")
- defineClass(name, bytes, 0, bytes.length)
- } else if (!isSharedClass(name)) {
- logDebug(s"hive class: $name - ${getResource(classToPath(name))}")
- super.loadClass(name, resolve)
- } else {
- // For shared classes, we delegate to baseClassLoader.
- logDebug(s"shared class: $name")
- baseClassLoader.loadClass(name)
+ def doLoadClass(name: String, resolve: Boolean): Class[_] = {
+ val classFileName = name.replaceAll("\\.", "/") + ".class"
+ if (isBarrierClass(name)) {
+ // For barrier classes, we construct a new copy of the class.
+ val bytes = IOUtils.toByteArray(baseClassLoader.getResourceAsStream(classFileName))
+ logDebug(s"custom defining: $name - ${util.Arrays.hashCode(bytes)}")
+ defineClass(name, bytes, 0, bytes.length)
+ } else if (!isSharedClass(name)) {
+ logDebug(s"hive class: $name - ${getResource(classToPath(name))}")
+ super.loadClass(name, resolve)
+ } else {
+ // For shared classes, we delegate to baseClassLoader.
+ logDebug(s"shared class: $name")
+ baseClassLoader.loadClass(name)
+ }
}
}
+ } else {
+ baseClassLoader
}
- // Pre-reflective instantiation setup.
- logDebug("Initializing the logger to avoid disaster...")
- Thread.currentThread.setContextClassLoader(classLoader)
+ private[hive] def addJar(path: String): Unit = synchronized {
+ val jarURL = new java.io.File(path).toURI.toURL
+ classLoader = new java.net.URLClassLoader(Array(jarURL), classLoader)
--- End diff --
While you are changing this part, can you add a TODO at here saying something like we should avoid of stacking classloaders (it will be good to use a single URLClassLoader and add jars to that).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146365916
Cool. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SQL] Improve session management...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144903112
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SQL] Improve session management...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144861238
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SQL] Improve session management...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144928539
[Test build #43169 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43169/console) for PR 8909 at commit [`3b48dd8`](https://github.com/apache/spark/commit/3b48dd8c9d3beb28c03dd6cdf6cf9e629a1645fd).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` /** Run a function within Hive state (SessionState, HiveConf, Hive client and class loader) */`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146386234
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43350/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SQL] Improve session management...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144814695
[Test build #43158 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43158/consoleFull) for PR 8909 at commit [`69f87cb`](https://github.com/apache/spark/commit/69f87cb1bd7d243a7a2e8e6eccd45c25845efac1).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/8909
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146715405
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43428/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SQL] Improve session management...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144928608
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43169/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r40389635
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala ---
@@ -68,6 +70,14 @@ class SimpleFunctionRegistry extends FunctionRegistry {
override def lookupFunction(name: String): Option[ExpressionInfo] = {
functionBuilders.get(name).map(_._1)
}
+
+ override def copy(): SimpleFunctionRegistry = {
--- End diff --
OK this is technically not part of this, but since you are modifying it ... shouldn't all of these catalogs be thread safe? Since they can be called by multiple threads.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41420832
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -1208,36 +1204,72 @@ object SQLContext {
private val INSTANTIATION_LOCK = new Object()
/**
- * Reference to the last created SQLContext.
+ * The active SQLContext for threads.
+ */
+ private val activeContexts: InheritableThreadLocal[SQLContext] =
--- End diff --
Nit: I think this should be singular since there is only one active context per thread.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146682900
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-143072264
[Test build #42988 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42988/consoleFull) for PR 8909 at commit [`0cf26a5`](https://github.com/apache/spark/commit/0cf26a57c844bd7379e07f518555418b0b3ecefb).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r40385814
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala ---
@@ -68,6 +70,14 @@ class SimpleFunctionRegistry extends FunctionRegistry {
override def lookupFunction(name: String): Option[ExpressionInfo] = {
functionBuilders.get(name).map(_._1)
}
+
+ override def copy(): SimpleFunctionRegistry = {
--- End diff --
should this be synchronized?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by WangTaoTheTonic <gi...@git.apache.org>.
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146812382
Nice work! As it is a blocker could we merge this into branch-1.5 too?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SQL] Improve session management...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144875601
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SQL] Improve session management...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144896030
[Test build #43165 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43165/console) for PR 8909 at commit [`72e137d`](https://github.com/apache/spark/commit/72e137d146db9e77efe0d64bfa5bb909d16654c2).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` /** Run a function within Hive state (SessionState, HiveConf, Hive client and class loader) */`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41456305
--- Diff: sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala ---
@@ -210,6 +190,9 @@ private[hive] class SparkExecuteStatementOperation(
statementId = UUID.randomUUID().toString
logInfo(s"Running query '$statement' with $statementId")
setState(OperationState.RUNNING)
+ val executionHiveClassLoader =
+ hiveContext.executionHive.state.getConf.getClassLoader
+ Thread.currentThread().setContextClassLoader(executionHiveClassLoader)
--- End diff --
Would be nice to add a comment here to explain why we need to set the class loader here. (To make `add jar` work properly right?)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144490411
[Test build #43135 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43135/consoleFull) for PR 8909 at commit [`5160fea`](https://github.com/apache/spark/commit/5160fea483e7a3b224b20aef01cb371898c0d5cf).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SQL] Improve session management...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144903860
[Test build #43169 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43169/consoleFull) for PR 8909 at commit [`3b48dd8`](https://github.com/apache/spark/commit/3b48dd8c9d3beb28c03dd6cdf6cf9e629a1645fd).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SQL] Improve session management...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144816852
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SQL] Improve session management...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144864855
[Test build #43162 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43162/consoleFull) for PR 8909 at commit [`9b82c55`](https://github.com/apache/spark/commit/9b82c55090080ca34b2b9e271c08afdeb0dd25e6).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146370212
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43356/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146372475
[Test build #1856 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1856/consoleFull) for PR 8909 at commit [`3aeae47`](https://github.com/apache/spark/commit/3aeae474fdada66b38b8ef81aa1523880f1b8a8a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-143056435
[Test build #42982 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42982/consoleFull) for PR 8909 at commit [`bc9f064`](https://github.com/apache/spark/commit/bc9f06447482dac5cfc61fd9d8c21bacb2300431).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41421569
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -1208,36 +1204,72 @@ object SQLContext {
private val INSTANTIATION_LOCK = new Object()
/**
- * Reference to the last created SQLContext.
+ * The active SQLContext for threads.
+ */
+ private val activeContexts: InheritableThreadLocal[SQLContext] =
+ new InheritableThreadLocal[SQLContext]
+
+ /**
+ * Reference to the created SQLContext.
*/
- @transient private val lastInstantiatedContext = new AtomicReference[SQLContext]()
+ @transient private val instantiatedContext = new AtomicReference[SQLContext]()
/**
* Get the singleton SQLContext if it exists or create a new one using the given SparkContext.
+ *
* This function can be used to create a singleton SQLContext object that can be shared across
* the JVM.
+ *
+ * If there an active SQLContext for current thread, it will returned instead of the global one.
*/
def getOrCreate(sparkContext: SparkContext): SQLContext = {
+ val ctx = activeContexts.get()
+ if (ctx != null) {
+ return ctx
+ }
+
INSTANTIATION_LOCK.synchronized {
- if (lastInstantiatedContext.get() == null) {
+ val ctx = instantiatedContext.get()
+ if (ctx == null) {
new SQLContext(sparkContext)
+ } else {
+ ctx
}
}
- lastInstantiatedContext.get()
}
- private[sql] def clearLastInstantiatedContext(): Unit = {
+ private[sql] def clearInstantiatedContext(): Unit = {
+ INSTANTIATION_LOCK.synchronized {
+ instantiatedContext.set(null)
+ }
+ }
+
+ private[sql] def clearTheInstantiatedContext(sqlContext: SQLContext): Unit = {
INSTANTIATION_LOCK.synchronized {
- lastInstantiatedContext.set(null)
+ instantiatedContext.compareAndSet(sqlContext, null)
}
}
- private[sql] def setLastInstantiatedContext(sqlContext: SQLContext): Unit = {
+ private[sql] def setInstantiatedContext(sqlContext: SQLContext): Unit = {
INSTANTIATION_LOCK.synchronized {
- lastInstantiatedContext.set(sqlContext)
+ instantiatedContext.compareAndSet(null, sqlContext)
}
}
+ /*
+ * Set a SQLContext for current thread
--- End diff --
"Changes the SQLContext that will be returned in this thread and its children when `SQLContext.getOrCreate()` is called. This can be used to ensure that a given thread receives a SQLContext with an isolated session, instead of the global (first created) context."
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146633053
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41422209
--- Diff: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala ---
@@ -289,16 +293,15 @@ class HiveThriftBinaryServerSuite extends HiveThriftJdbcTest {
rs2.close()
},
- // accessing the cached data in another session
+ // try to access the cached data in another session
{ statement =>
- val rs1 = statement.executeQuery("SELECT key FROM test_table ORDER BY KEY DESC")
- val buf1 = new collection.mutable.ArrayBuffer[Int]()
- while (rs1.next()) {
- buf1 += rs1.getInt(1)
+ // Cached temporary table can't be accessed by other sessions
+ intercept[SQLException] {
+ statement.executeQuery("SELECT key FROM test_table ORDER BY KEY DESC")
}
- rs1.close()
+ // TODO need to figure out how to determine if the data loaded from cache
--- End diff --
You can grab the queryExecution.executedPlan and look for `InMemoryColumnarTableScan`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41447708
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala ---
@@ -51,22 +51,42 @@ class SimpleFunctionRegistry extends FunctionRegistry {
private val functionBuilders =
StringKeyHashMap[(ExpressionInfo, FunctionBuilder)](caseSensitive = false)
- override def registerFunction(name: String, info: ExpressionInfo, builder: FunctionBuilder)
- : Unit = {
- functionBuilders.put(name, (info, builder))
+ override def registerFunction(
+ name: String,
+ info: ExpressionInfo,
+ builder: FunctionBuilder): Unit = {
+ synchronized {
--- End diff --
In copy(), `iterator.foreach` can be protected by the synchronization of map.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146898039
@WangTaoTheTonic Since this is a not a bug fix (instead, a major new feature), we will not merge it to branch-1.5. It will be released with Spark 1.6.0.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41456081
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala ---
@@ -1133,6 +1133,17 @@ class HiveQuerySuite extends HiveComparisonTest with BeforeAndAfter {
conf.clear()
}
+ test("current_database with mutiple sessions") {
+ sql("create database a")
+ sql("use a")
+ val s2 = newSession()
+ s2.sql("create database b")
+ s2.sql("use b")
+
+ assert(sql("select current_database()").first() === Row("a"))
+ assert(s2.sql("select current_database()").first() === Row("b"))
--- End diff --
Added.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41542722
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -1196,49 +1174,101 @@ class SQLContext(@transient val sparkContext: SparkContext)
// Register a succesfully instantiatd context to the singleton. This should be at the end of
// the class definition so that the singleton is updated only if there is no exception in the
// construction of the instance.
- SQLContext.setLastInstantiatedContext(self)
+ sparkContext.addSparkListener(new SparkListener{
+ override def onApplicationEnd(applicationEnd: SparkListenerApplicationEnd) {
--- End diff --
This is implicit `Unit`, but I think they are going to get rid of this in a future version of scala (and I think its against our style guide).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-143071388
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SQL] Improve session management...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144862827
[Test build #43160 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43160/console) for PR 8909 at commit [`d10a97a`](https://github.com/apache/spark/commit/d10a97a23608e2f27bbeb6760559972b26d64c56).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` /** Run a function within Hive state (SessionState, HiveConf, Hive client and class loader) */`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r40386226
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala ---
@@ -68,6 +70,14 @@ class SimpleFunctionRegistry extends FunctionRegistry {
override def lookupFunction(name: String): Option[ExpressionInfo] = {
functionBuilders.get(name).map(_._1)
}
+
+ override def copy(): SimpleFunctionRegistry = {
--- End diff --
SimpleFunctionRegistry is not designed to be used by multiple threads (read/write in the same time), I think it's fine now, because the builtin is constructed in the begging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41427708
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala ---
@@ -148,53 +150,88 @@ private[hive] class IsolatedClientLoader(
name.replaceAll("\\.", "/") + ".class"
/** The classloader that is used to load an isolated version of Hive. */
- protected val classLoader: ClassLoader = new URLClassLoader(allJars, rootClassLoader) {
- override def loadClass(name: String, resolve: Boolean): Class[_] = {
- val loaded = findLoadedClass(name)
- if (loaded == null) doLoadClass(name, resolve) else loaded
- }
+ private[hive] var classLoader: ClassLoader = if (isolationOn) {
+ new URLClassLoader(allJars, rootClassLoader) {
--- End diff --
Since we are overriding more stuff, should we have a explicit class definition with docs about how it works and reasons that we need it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41457829
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala ---
@@ -264,12 +262,16 @@ private[hive] object HiveQl extends Logging {
* Returns the HiveConf
*/
private[this] def hiveConf: HiveConf = {
- val ss = SessionState.get() // SessionState is lazy initialization, it can be null here
+ var ss = SessionState.get()
+ // SessionState is lazy initialization, it can be null here
if (ss == null) {
- new HiveConf()
- } else {
- ss.getConf
+ val original = Thread.currentThread().getContextClassLoader
+ val conf = new HiveConf(classOf[SessionState])
+ conf.setClassLoader(original)
+ ss = new SessionState(conf)
+ SessionState.start(ss)
--- End diff --
Actually, I guess we are fine. When we create a `HiveConf`, its internal classloader will be `Thread.currentThread().getContextClassLoader()`. So, we will not change the context classloader. But, it is better to have explicitly call `conf.setClassLoader` to set `original` as the classloader for this conf to avoid any confusion. What do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SQL] Improve session management...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144867432
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144797066
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SQL] Improve session management...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144867392
[Test build #43162 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43162/console) for PR 8909 at commit [`9b82c55`](https://github.com/apache/spark/commit/9b82c55090080ca34b2b9e271c08afdeb0dd25e6).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` /** Run a function within Hive state (SessionState, HiveConf, Hive client and class loader) */`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146386233
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SQL] Improve session management...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144861265
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SQL] Improve session management...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144812918
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SQL] Improve session management...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144862829
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41425289
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -1208,36 +1204,72 @@ object SQLContext {
private val INSTANTIATION_LOCK = new Object()
/**
- * Reference to the last created SQLContext.
+ * The active SQLContext for threads.
+ */
+ private val activeContexts: InheritableThreadLocal[SQLContext] =
+ new InheritableThreadLocal[SQLContext]
+
+ /**
+ * Reference to the created SQLContext.
*/
- @transient private val lastInstantiatedContext = new AtomicReference[SQLContext]()
+ @transient private val instantiatedContext = new AtomicReference[SQLContext]()
/**
* Get the singleton SQLContext if it exists or create a new one using the given SparkContext.
+ *
* This function can be used to create a singleton SQLContext object that can be shared across
* the JVM.
+ *
+ * If there an active SQLContext for current thread, it will returned instead of the global one.
*/
def getOrCreate(sparkContext: SparkContext): SQLContext = {
+ val ctx = activeContexts.get()
+ if (ctx != null) {
+ return ctx
+ }
+
INSTANTIATION_LOCK.synchronized {
- if (lastInstantiatedContext.get() == null) {
+ val ctx = instantiatedContext.get()
+ if (ctx == null) {
new SQLContext(sparkContext)
+ } else {
+ ctx
}
}
- lastInstantiatedContext.get()
}
- private[sql] def clearLastInstantiatedContext(): Unit = {
+ private[sql] def clearInstantiatedContext(): Unit = {
+ INSTANTIATION_LOCK.synchronized {
+ instantiatedContext.set(null)
+ }
+ }
+
+ private[sql] def clearTheInstantiatedContext(sqlContext: SQLContext): Unit = {
INSTANTIATION_LOCK.synchronized {
- lastInstantiatedContext.set(null)
+ instantiatedContext.compareAndSet(sqlContext, null)
}
}
- private[sql] def setLastInstantiatedContext(sqlContext: SQLContext): Unit = {
+ private[sql] def setInstantiatedContext(sqlContext: SQLContext): Unit = {
INSTANTIATION_LOCK.synchronized {
- lastInstantiatedContext.set(sqlContext)
+ instantiatedContext.compareAndSet(null, sqlContext)
}
}
+ /*
+ * Set a SQLContext for current thread
+ */
+ def setActive(ctx: SQLContext): Unit = {
+ activeContexts.set(ctx)
+ }
+
+ /**
+ * Clear the active SQLContext for current thread
+ */
+ def clearActive(): Unit = {
+ activeContexts.remove()
+ }
+
--- End diff --
Can we have a section in the scaladoc to give an overview of these newly added methods?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146006628
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SQL] Improve session management...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144799919
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41462692
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -1196,49 +1174,101 @@ class SQLContext(@transient val sparkContext: SparkContext)
// Register a succesfully instantiatd context to the singleton. This should be at the end of
// the class definition so that the singleton is updated only if there is no exception in the
// construction of the instance.
- SQLContext.setLastInstantiatedContext(self)
+ sparkContext.addSparkListener(new SparkListener{
+ override def onApplicationEnd(applicationEnd: SparkListenerApplicationEnd) {
+ SQLContext.clearTheInstantiatedContext(self)
+ }
+ })
+
+ SQLContext.setInstantiatedContext(self)
}
/**
* This SQLContext object contains utility functions to create a singleton SQLContext instance,
- * or to get the last created SQLContext instance.
+ * or to get the created SQLContext instance.
*/
object SQLContext {
private val INSTANTIATION_LOCK = new Object()
/**
- * Reference to the last created SQLContext.
+ * The active SQLContext for the current thread.
*/
- @transient private val lastInstantiatedContext = new AtomicReference[SQLContext]()
+ private val activeContext: InheritableThreadLocal[SQLContext] =
+ new InheritableThreadLocal[SQLContext]
+
+ /**
+ * Reference to the created SQLContext.
+ */
+ @transient private val instantiatedContext = new AtomicReference[SQLContext]()
--- End diff --
is it possible to have just one of these? It seems currently that we only use `activeContext` if the user explicitly called `setActive`. Can we just use the same `AtomicReference` for `setActive`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144568839
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r42816917
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -1196,46 +1174,90 @@ class SQLContext(@transient val sparkContext: SparkContext)
// Register a succesfully instantiatd context to the singleton. This should be at the end of
// the class definition so that the singleton is updated only if there is no exception in the
// construction of the instance.
- SQLContext.setLastInstantiatedContext(self)
+ sparkContext.addSparkListener(new SparkListener {
+ override def onApplicationEnd(applicationEnd: SparkListenerApplicationEnd): Unit = {
+ SQLContext.clearInstantiatedContext(self)
+ }
+ })
+
+ SQLContext.setInstantiatedContext(self)
}
/**
* This SQLContext object contains utility functions to create a singleton SQLContext instance,
- * or to get the last created SQLContext instance.
+ * or to get the created SQLContext instance.
+ *
+ * It also provides utility functions to support preference for threads in multiple sessions
+ * scenario, setActive could set a SQLContext for current thread, which will be returned by
+ * getOrCreate instead of the global one.
*/
object SQLContext {
- private val INSTANTIATION_LOCK = new Object()
+ /**
+ * The active SQLContext for the current thread.
+ */
+ private val activeContext: InheritableThreadLocal[SQLContext] =
+ new InheritableThreadLocal[SQLContext]
/**
- * Reference to the last created SQLContext.
+ * Reference to the created SQLContext.
*/
- @transient private val lastInstantiatedContext = new AtomicReference[SQLContext]()
+ @transient private val instantiatedContext = new AtomicReference[SQLContext]()
--- End diff --
isn't AtomicReference itself just a wrapper around volatile field?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146349780
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41464483
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -1196,49 +1174,101 @@ class SQLContext(@transient val sparkContext: SparkContext)
// Register a succesfully instantiatd context to the singleton. This should be at the end of
// the class definition so that the singleton is updated only if there is no exception in the
// construction of the instance.
- SQLContext.setLastInstantiatedContext(self)
+ sparkContext.addSparkListener(new SparkListener{
+ override def onApplicationEnd(applicationEnd: SparkListenerApplicationEnd) {
+ SQLContext.clearTheInstantiatedContext(self)
+ }
+ })
+
+ SQLContext.setInstantiatedContext(self)
}
/**
* This SQLContext object contains utility functions to create a singleton SQLContext instance,
- * or to get the last created SQLContext instance.
+ * or to get the created SQLContext instance.
*/
object SQLContext {
private val INSTANTIATION_LOCK = new Object()
/**
- * Reference to the last created SQLContext.
+ * The active SQLContext for the current thread.
*/
- @transient private val lastInstantiatedContext = new AtomicReference[SQLContext]()
+ private val activeContext: InheritableThreadLocal[SQLContext] =
+ new InheritableThreadLocal[SQLContext]
+
+ /**
+ * Reference to the created SQLContext.
+ */
+ @transient private val instantiatedContext = new AtomicReference[SQLContext]()
/**
* Get the singleton SQLContext if it exists or create a new one using the given SparkContext.
+ *
* This function can be used to create a singleton SQLContext object that can be shared across
* the JVM.
+ *
+ * If there is an active SQLContext for current thread, it will returned instead of the global
+ * one.
+ *
+ * @since 1.5.0
*/
def getOrCreate(sparkContext: SparkContext): SQLContext = {
+ val ctx = activeContext.get()
+ if (ctx != null) {
+ return ctx
+ }
+
INSTANTIATION_LOCK.synchronized {
- if (lastInstantiatedContext.get() == null) {
+ val ctx = instantiatedContext.get()
+ if (ctx == null) {
new SQLContext(sparkContext)
+ } else {
+ ctx
}
}
- lastInstantiatedContext.get()
}
- private[sql] def clearLastInstantiatedContext(): Unit = {
+ private[sql] def clearInstantiatedContext(): Unit = {
--- End diff --
this isn't used anymore except in tests is it? Do we still need it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144488609
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41426540
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala ---
@@ -69,13 +84,24 @@ private[hive] class HiveQLDialect extends ParserDialect {
*
* @since 1.0.0
*/
-class HiveContext(sc: SparkContext) extends SQLContext(sc) with Logging {
+class HiveContext private[hive](
+ sc: SparkContext,
+ cacheManager: CacheManager,
+ @transient execHive: ClientWrapper,
+ @transient metaHive: ClientInterface) extends SQLContext(sc, cacheManager) with Logging {
--- End diff --
Maybe add comments to explain what `execHive` and `metaHive` do?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41427065
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala ---
@@ -136,7 +137,8 @@ private[hive] class IsolatedClientLoader(
(name.startsWith("com.google") && !name.startsWith("com.google.cloud")) ||
name.startsWith("java.lang.") ||
name.startsWith("java.net") ||
- sharedPrefixes.exists(name.startsWith)
+ sharedPrefixes.exists(name.startsWith) ||
+ name.startsWith("[")
--- End diff --
Since it is not super obvious, can we add a comment to say that this is for arrays?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41423533
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/test/TestHive.scala ---
@@ -116,27 +116,19 @@ class TestHiveContext(sc: SparkContext) extends HiveContext(sc) {
override def executePlan(plan: LogicalPlan): this.QueryExecution =
new this.QueryExecution(plan)
- // Make sure we set those test specific confs correctly when we create
- // the SQLConf as well as when we call clear.
- override protected[sql] def createSession(): SQLSession = {
- new this.SQLSession()
- }
-
- protected[hive] class SQLSession extends super.SQLSession {
- protected[sql] override lazy val conf: SQLConf = new SQLConf {
- // TODO as in unit test, conf.clear() probably be called, all of the value will be cleared.
- // The super.getConf(SQLConf.DIALECT) is "sql" by default, we need to set it as "hiveql"
- override def dialect: String = super.getConf(SQLConf.DIALECT, "hiveql")
- override def caseSensitiveAnalysis: Boolean = getConf(SQLConf.CASE_SENSITIVE, false)
+ protected[sql] override lazy val conf: SQLConf = new SQLConf {
+ // TODO as in unit test, conf.clear() probably be called, all of the value will be cleared.
--- End diff --
existing: is this TODO relevant still?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41428095
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala ---
@@ -148,53 +150,88 @@ private[hive] class IsolatedClientLoader(
name.replaceAll("\\.", "/") + ".class"
/** The classloader that is used to load an isolated version of Hive. */
- protected val classLoader: ClassLoader = new URLClassLoader(allJars, rootClassLoader) {
- override def loadClass(name: String, resolve: Boolean): Class[_] = {
- val loaded = findLoadedClass(name)
- if (loaded == null) doLoadClass(name, resolve) else loaded
- }
+ private[hive] var classLoader: ClassLoader = if (isolationOn) {
+ new URLClassLoader(allJars, rootClassLoader) {
+ val cache = new java.util.concurrent.ConcurrentHashMap[String, Class[_]]
--- End diff --
If we want to use it as a cache, should we use a data structure that defines the cache capacity?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-143089407
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146365434
@yhuai UDF and UDTF are also covered by testes, also checked that manually, work fine.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146351112
[Test build #43350 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43350/consoleFull) for PR 8909 at commit [`b70720f`](https://github.com/apache/spark/commit/b70720fa18fc6361d3ea9d394a32c2c603b2018f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41420988
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -1208,36 +1204,72 @@ object SQLContext {
private val INSTANTIATION_LOCK = new Object()
/**
- * Reference to the last created SQLContext.
+ * The active SQLContext for threads.
+ */
+ private val activeContexts: InheritableThreadLocal[SQLContext] =
+ new InheritableThreadLocal[SQLContext]
+
+ /**
+ * Reference to the created SQLContext.
*/
- @transient private val lastInstantiatedContext = new AtomicReference[SQLContext]()
+ @transient private val instantiatedContext = new AtomicReference[SQLContext]()
--- End diff --
I think this makes more sense than "last" but we should update the objects doc if we are going to change the semantics.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144525512
[Test build #43135 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43135/console) for PR 8909 at commit [`5160fea`](https://github.com/apache/spark/commit/5160fea483e7a3b224b20aef01cb371898c0d5cf).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class HyperLogLogPlusPlus(child: Expression, relativeSD: Double = 0.05)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41456023
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala ---
@@ -264,12 +262,16 @@ private[hive] object HiveQl extends Logging {
* Returns the HiveConf
*/
private[this] def hiveConf: HiveConf = {
- val ss = SessionState.get() // SessionState is lazy initialization, it can be null here
+ var ss = SessionState.get()
+ // SessionState is lazy initialization, it can be null here
if (ss == null) {
- new HiveConf()
- } else {
- ss.getConf
+ val original = Thread.currentThread().getContextClassLoader
+ val conf = new HiveConf(classOf[SessionState])
+ conf.setClassLoader(original)
+ ss = new SessionState(conf)
+ SessionState.start(ss)
--- End diff --
In the case of no SessionState for current thread, it will be NPE if we do not call `start()`. Should this only used by tests?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by tedyu <gi...@git.apache.org>.
Github user tedyu commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r42817261
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -1196,46 +1174,90 @@ class SQLContext(@transient val sparkContext: SparkContext)
// Register a succesfully instantiatd context to the singleton. This should be at the end of
// the class definition so that the singleton is updated only if there is no exception in the
// construction of the instance.
- SQLContext.setLastInstantiatedContext(self)
+ sparkContext.addSparkListener(new SparkListener {
+ override def onApplicationEnd(applicationEnd: SparkListenerApplicationEnd): Unit = {
+ SQLContext.clearInstantiatedContext(self)
+ }
+ })
+
+ SQLContext.setInstantiatedContext(self)
}
/**
* This SQLContext object contains utility functions to create a singleton SQLContext instance,
- * or to get the last created SQLContext instance.
+ * or to get the created SQLContext instance.
+ *
+ * It also provides utility functions to support preference for threads in multiple sessions
+ * scenario, setActive could set a SQLContext for current thread, which will be returned by
+ * getOrCreate instead of the global one.
*/
object SQLContext {
- private val INSTANTIATION_LOCK = new Object()
+ /**
+ * The active SQLContext for the current thread.
+ */
+ private val activeContext: InheritableThreadLocal[SQLContext] =
+ new InheritableThreadLocal[SQLContext]
/**
- * Reference to the last created SQLContext.
+ * Reference to the created SQLContext.
*/
- @transient private val lastInstantiatedContext = new AtomicReference[SQLContext]()
+ @transient private val instantiatedContext = new AtomicReference[SQLContext]()
--- End diff --
Alright.
I was looking at getOrCreate() at line 1219
Seems the access to instantiatedContext is proper.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41424458
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala ---
@@ -410,7 +436,11 @@ class HiveContext(sc: SparkContext) extends SQLContext(sc) with Logging {
// Note that HiveUDFs will be overridden by functions registered in this context.
@transient
override protected[sql] lazy val functionRegistry: FunctionRegistry =
- new HiveFunctionRegistry(FunctionRegistry.builtin)
+ new HiveFunctionRegistry(FunctionRegistry.builtin.copy())
+
+ functionRegistry.registerFunction(
+ "current_database",
+ (expressions: Seq[Expression]) => new CurrentDatabase(this))
--- End diff --
This is a cute way to tie this to the session.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146366859
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-143073095
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42988/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SQL] Improve session management...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144831719
[Test build #43155 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43155/console) for PR 8909 at commit [`9fe30cf`](https://github.com/apache/spark/commit/9fe30cf6372fd5c78a6aae004e19bd038f1cc93f).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146349700
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146370148
[Test build #43356 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43356/console) for PR 8909 at commit [`e16737c`](https://github.com/apache/spark/commit/e16737c18f2662da1ddfd00feb9e5f433ec94184).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` /** Run a function within Hive state (SessionState, HiveConf, Hive client and class loader) */`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144529838
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41421063
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -1208,36 +1204,72 @@ object SQLContext {
private val INSTANTIATION_LOCK = new Object()
/**
- * Reference to the last created SQLContext.
+ * The active SQLContext for threads.
+ */
+ private val activeContexts: InheritableThreadLocal[SQLContext] =
+ new InheritableThreadLocal[SQLContext]
+
+ /**
+ * Reference to the created SQLContext.
*/
- @transient private val lastInstantiatedContext = new AtomicReference[SQLContext]()
+ @transient private val instantiatedContext = new AtomicReference[SQLContext]()
/**
* Get the singleton SQLContext if it exists or create a new one using the given SparkContext.
+ *
* This function can be used to create a singleton SQLContext object that can be shared across
* the JVM.
+ *
+ * If there an active SQLContext for current thread, it will returned instead of the global one.
*/
def getOrCreate(sparkContext: SparkContext): SQLContext = {
+ val ctx = activeContexts.get()
+ if (ctx != null) {
+ return ctx
+ }
+
INSTANTIATION_LOCK.synchronized {
- if (lastInstantiatedContext.get() == null) {
+ val ctx = instantiatedContext.get()
+ if (ctx == null) {
new SQLContext(sparkContext)
+ } else {
+ ctx
}
}
- lastInstantiatedContext.get()
}
- private[sql] def clearLastInstantiatedContext(): Unit = {
+ private[sql] def clearInstantiatedContext(): Unit = {
+ INSTANTIATION_LOCK.synchronized {
+ instantiatedContext.set(null)
+ }
+ }
+
+ private[sql] def clearTheInstantiatedContext(sqlContext: SQLContext): Unit = {
INSTANTIATION_LOCK.synchronized {
- lastInstantiatedContext.set(null)
+ instantiatedContext.compareAndSet(sqlContext, null)
}
}
- private[sql] def setLastInstantiatedContext(sqlContext: SQLContext): Unit = {
+ private[sql] def setInstantiatedContext(sqlContext: SQLContext): Unit = {
INSTANTIATION_LOCK.synchronized {
- lastInstantiatedContext.set(sqlContext)
+ instantiatedContext.compareAndSet(null, sqlContext)
}
}
+ /*
--- End diff --
nit: indent and need two *
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41422069
--- Diff: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala ---
@@ -159,7 +159,7 @@ class CliSuite extends SparkFunSuite with BeforeAndAfter with Logging {
s"LOAD DATA LOCAL INPATH '$dataFilePath' OVERWRITE INTO TABLE hive_test;"
-> "OK",
"CACHE TABLE hive_test;"
- -> "Time taken: ",
+ -> "",
--- End diff --
What's the reason for these changes?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41424399
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -64,18 +63,28 @@ import org.apache.spark.util.Utils
*
* @since 1.0.0
*/
-class SQLContext(@transient val sparkContext: SparkContext)
- extends org.apache.spark.Logging
- with Serializable {
+class SQLContext private[sql](
+ @transient val sparkContext: SparkContext,
+ @transient protected[sql] val cacheManager: CacheManager)
+ extends org.apache.spark.Logging with Serializable {
self =>
+ def this(sparkContext: SparkContext) = this(sparkContext, new CacheManager)
--- End diff --
Let's add scaladoc to explain when it is recommended to use this constructor and when it recommended to use `newSession`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146367974
[Test build #43357 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43357/consoleFull) for PR 8909 at commit [`3aeae47`](https://github.com/apache/spark/commit/3aeae474fdada66b38b8ef81aa1523880f1b8a8a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SQL] Improve session management...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144876011
[Test build #43165 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43165/consoleFull) for PR 8909 at commit [`72e137d`](https://github.com/apache/spark/commit/72e137d146db9e77efe0d64bfa5bb909d16654c2).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-143078365
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41450974
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala ---
@@ -136,7 +137,8 @@ private[hive] class IsolatedClientLoader(
(name.startsWith("com.google") && !name.startsWith("com.google.cloud")) ||
name.startsWith("java.lang.") ||
name.startsWith("java.net") ||
- sharedPrefixes.exists(name.startsWith)
+ sharedPrefixes.exists(name.startsWith) ||
+ name.startsWith("[")
--- End diff --
Thanks for the explanation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-143079274
[Test build #42991 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42991/consoleFull) for PR 8909 at commit [`3723768`](https://github.com/apache/spark/commit/372376862ccbf1877dbbb2a26761c76d4b3afdd5).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146366850
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SQL] Improve session management...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41070271
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -300,21 +303,25 @@ class SQLContext(@transient val sparkContext: SparkContext)
* @group cachemgmt
* @since 1.3.0
*/
- def isCached(tableName: String): Boolean = cacheManager.isCached(tableName)
+ def isCached(tableName: String): Boolean = {
+ cacheManager.lookupCachedData(table(tableName)).nonEmpty
+ }
/**
* Caches the specified table in-memory.
* @group cachemgmt
* @since 1.3.0
*/
- def cacheTable(tableName: String): Unit = cacheManager.cacheTable(tableName)
+ def cacheTable(tableName: String): Unit = {
+ cacheManager.cacheQuery(this, table(tableName), Some(tableName))
+ }
/**
* Removes the specified table from the in-memory cache.
* @group cachemgmt
* @since 1.3.0
*/
- def uncacheTable(tableName: String): Unit = cacheManager.uncacheTable(tableName)
+ def uncacheTable(tableName: String): Unit = cacheManager.uncacheQuery(table(tableName))
--- End diff --
We can also add `cacheQuery` here so that we don't need to pass in a `SQLContext` every time we call `CacheManager.cacheQuery `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by Neuw84 <gi...@git.apache.org>.
Github user Neuw84 commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-175916186
@deenar, I saw it after many hours reading the code in the web docs . Although, I think that the HiveSparkContext should implement the same logic as the SparkContext where you can get the same Session programmatically. Thanks by the way
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41427431
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala ---
@@ -136,7 +137,8 @@ private[hive] class IsolatedClientLoader(
(name.startsWith("com.google") && !name.startsWith("com.google.cloud")) ||
name.startsWith("java.lang.") ||
name.startsWith("java.net") ||
- sharedPrefixes.exists(name.startsWith)
+ sharedPrefixes.exists(name.startsWith) ||
+ name.startsWith("[")
--- End diff --
Haha, yeah that took me a second. Any idea why this is needed now? (it seems reasonable to add).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41303637
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -300,21 +303,25 @@ class SQLContext(@transient val sparkContext: SparkContext)
* @group cachemgmt
* @since 1.3.0
*/
- def isCached(tableName: String): Boolean = cacheManager.isCached(tableName)
+ def isCached(tableName: String): Boolean = {
+ cacheManager.lookupCachedData(table(tableName)).nonEmpty
+ }
/**
* Caches the specified table in-memory.
* @group cachemgmt
* @since 1.3.0
*/
- def cacheTable(tableName: String): Unit = cacheManager.cacheTable(tableName)
+ def cacheTable(tableName: String): Unit = {
+ cacheManager.cacheQuery(this, table(tableName), Some(tableName))
+ }
/**
* Removes the specified table from the in-memory cache.
* @group cachemgmt
* @since 1.3.0
*/
- def uncacheTable(tableName: String): Unit = cacheManager.uncacheTable(tableName)
+ def uncacheTable(tableName: String): Unit = cacheManager.uncacheQuery(table(tableName))
--- End diff --
done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144488636
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146367494
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41455821
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala ---
@@ -69,13 +84,24 @@ private[hive] class HiveQLDialect extends ParserDialect {
*
* @since 1.0.0
*/
-class HiveContext(sc: SparkContext) extends SQLContext(sc) with Logging {
+class HiveContext private[hive](
+ sc: SparkContext,
+ cacheManager: CacheManager,
+ @transient execHive: ClientWrapper,
+ @transient metaHive: ClientInterface) extends SQLContext(sc, cacheManager) with Logging {
--- End diff --
This is not public API. Those two are used to create a new session with existing clients (forked from another one).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146386181
[Test build #43350 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43350/console) for PR 8909 at commit [`b70720f`](https://github.com/apache/spark/commit/b70720fa18fc6361d3ea9d394a32c2c603b2018f).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` /** Run a function within Hive state (SessionState, HiveConf, Hive client and class loader) */`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41425044
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -1208,36 +1204,72 @@ object SQLContext {
private val INSTANTIATION_LOCK = new Object()
/**
- * Reference to the last created SQLContext.
+ * The active SQLContext for threads.
+ */
+ private val activeContexts: InheritableThreadLocal[SQLContext] =
+ new InheritableThreadLocal[SQLContext]
+
+ /**
+ * Reference to the created SQLContext.
*/
- @transient private val lastInstantiatedContext = new AtomicReference[SQLContext]()
+ @transient private val instantiatedContext = new AtomicReference[SQLContext]()
/**
* Get the singleton SQLContext if it exists or create a new one using the given SparkContext.
+ *
* This function can be used to create a singleton SQLContext object that can be shared across
* the JVM.
+ *
+ * If there an active SQLContext for current thread, it will returned instead of the global one.
*/
def getOrCreate(sparkContext: SparkContext): SQLContext = {
+ val ctx = activeContexts.get()
+ if (ctx != null) {
+ return ctx
+ }
+
INSTANTIATION_LOCK.synchronized {
- if (lastInstantiatedContext.get() == null) {
+ val ctx = instantiatedContext.get()
+ if (ctx == null) {
new SQLContext(sparkContext)
+ } else {
+ ctx
}
}
- lastInstantiatedContext.get()
}
- private[sql] def clearLastInstantiatedContext(): Unit = {
+ private[sql] def clearInstantiatedContext(): Unit = {
+ INSTANTIATION_LOCK.synchronized {
+ instantiatedContext.set(null)
+ }
+ }
+
+ private[sql] def clearTheInstantiatedContext(sqlContext: SQLContext): Unit = {
--- End diff --
Should we change the name of of `clearTheInstantiatedContext` to reflect its difference with `clearInstantiatedContext`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146370859
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41421639
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -1208,36 +1204,72 @@ object SQLContext {
private val INSTANTIATION_LOCK = new Object()
/**
- * Reference to the last created SQLContext.
+ * The active SQLContext for threads.
+ */
+ private val activeContexts: InheritableThreadLocal[SQLContext] =
+ new InheritableThreadLocal[SQLContext]
+
+ /**
+ * Reference to the created SQLContext.
*/
- @transient private val lastInstantiatedContext = new AtomicReference[SQLContext]()
+ @transient private val instantiatedContext = new AtomicReference[SQLContext]()
/**
* Get the singleton SQLContext if it exists or create a new one using the given SparkContext.
+ *
* This function can be used to create a singleton SQLContext object that can be shared across
* the JVM.
+ *
+ * If there an active SQLContext for current thread, it will returned instead of the global one.
*/
def getOrCreate(sparkContext: SparkContext): SQLContext = {
+ val ctx = activeContexts.get()
+ if (ctx != null) {
+ return ctx
+ }
+
INSTANTIATION_LOCK.synchronized {
- if (lastInstantiatedContext.get() == null) {
+ val ctx = instantiatedContext.get()
+ if (ctx == null) {
new SQLContext(sparkContext)
+ } else {
+ ctx
}
}
- lastInstantiatedContext.get()
}
- private[sql] def clearLastInstantiatedContext(): Unit = {
+ private[sql] def clearInstantiatedContext(): Unit = {
+ INSTANTIATION_LOCK.synchronized {
+ instantiatedContext.set(null)
+ }
+ }
+
+ private[sql] def clearTheInstantiatedContext(sqlContext: SQLContext): Unit = {
INSTANTIATION_LOCK.synchronized {
- lastInstantiatedContext.set(null)
+ instantiatedContext.compareAndSet(sqlContext, null)
}
}
- private[sql] def setLastInstantiatedContext(sqlContext: SQLContext): Unit = {
+ private[sql] def setInstantiatedContext(sqlContext: SQLContext): Unit = {
INSTANTIATION_LOCK.synchronized {
- lastInstantiatedContext.set(sqlContext)
+ instantiatedContext.compareAndSet(null, sqlContext)
}
}
+ /*
+ * Set a SQLContext for current thread
+ */
+ def setActive(ctx: SQLContext): Unit = {
+ activeContexts.set(ctx)
+ }
+
+ /**
+ * Clear the active SQLContext for current thread
--- End diff --
"Clears the active SQLContext for current thread. Subsequent calls to getOrCreate will return the first created context instead of a thread-local override."
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146668946
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144532879
[Test build #43137 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43137/consoleFull) for PR 8909 at commit [`e942e93`](https://github.com/apache/spark/commit/e942e9333274527665d9a72d4a9ceb14ae711854).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144798668
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144525594
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43135/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144529764
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/8909#discussion_r41420506
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -64,18 +63,28 @@ import org.apache.spark.util.Utils
*
* @since 1.0.0
*/
-class SQLContext(@transient val sparkContext: SparkContext)
- extends org.apache.spark.Logging
- with Serializable {
+class SQLContext private[sql](
+ @transient val sparkContext: SparkContext,
+ @transient protected[sql] val cacheManager: CacheManager)
+ extends org.apache.spark.Logging with Serializable {
self =>
+ def this(sparkContext: SparkContext) = this(sparkContext, new CacheManager)
def this(sparkContext: JavaSparkContext) = this(sparkContext.sc)
/**
+ * Returns a SQLContext as new session, with separated SQL configurations, temporary tables,
+ * registered functions, but share the same SparkContext and CacheManager.
--- End diff --
nit: "sharing"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by navis <gi...@git.apache.org>.
Github user navis commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146104351
@davies I love the idea removing thread-local, which caused headache for me doing #7118. One thing I like to see also fixed in here is that jdbc meta calls like "connection.getMetaData" is not working currently because it's executed via exec-hive, which sees dummy-derby metastore.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-143078377
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SPARK-10902] [SQL] Improve sess...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146367511
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SQL] Improve session management...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144816845
[Test build #43158 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43158/console) for PR 8909 at commit [`69f87cb`](https://github.com/apache/spark/commit/69f87cb1bd7d243a7a2e8e6eccd45c25845efac1).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` /** Run a function within Hive state (SessionState, HiveConf, Hive client and class loader) */`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [SQL] Improve session management...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-144875582
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10810] [WIP] [SQL] Improve session mana...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-143080746
[Test build #1814 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1814/console) for PR 8909 at commit [`0cf26a5`](https://github.com/apache/spark/commit/0cf26a57c844bd7379e07f518555418b0b3ecefb).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org