You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by xguo27 <gi...@git.apache.org> on 2015/11/09 03:01:05 UTC
[GitHub] spark pull request: [SPARK-11562][SQL] Provide user an option to i...
GitHub user xguo27 opened a pull request:
https://github.com/apache/spark/pull/9553
[SPARK-11562][SQL] Provide user an option to init SQLContext or HiveContext in spark shell
Introducing a boolean property 'spark.sql.hive.context' to turn HiveContext on and off as the default sqlContext type.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/xguo27/spark SPARK-11562
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9553.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9553
----
commit cb5892cdd605ec70586c0670ed19d924e0a8eade
Author: Xiu Guo <gu...@us.ibm.com>
Date: 2015-11-08T20:33:07Z
[SPARK-11562][SQL] Provide user an option to init SQLContext or HiveContext in spark shell
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11562][SQL] Provide option to switch Sq...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9553#issuecomment-166025617
**[Test build #48057 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48057/consoleFull)** for PR 9553 at commit [`969008e`](https://github.com/apache/spark/commit/969008ef85723eb35d4ab038c9158eb3ea12b031).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11562][SQL] Provide option to switch Sq...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9553#issuecomment-165955622
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48049/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11562][SQL] Provide option to switch Sq...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/9553#discussion_r48095511
--- Diff: repl/scala-2.10/src/main/scala/org/apache/spark/repl/SparkILoop.scala ---
@@ -132,6 +132,7 @@ class SparkILoop(
@DeveloperApi
var sparkContext: SparkContext = _
var sqlContext: SQLContext = _
+ var useHiveContext: Boolean = _
--- End diff --
Can we make it a local field in `createSQLContext`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11562][SQL] Provide option to switch Sq...
Posted by xguo27 <gi...@git.apache.org>.
Github user xguo27 commented on the pull request:
https://github.com/apache/spark/pull/9553#issuecomment-162964945
Hi @yhuai, do you think this is good to merge?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11562][SQL] Provide option to switch Sq...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/9553#issuecomment-218835567
@xguo27 HiveContext is now removed and SQLContext is no longer the default thing. The spark-shell now uses `SparkSession` instead. The changes here are mostly outdated so let's close this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11562][SQL] Provide option to switch Sq...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/9553#issuecomment-165295242
@xguo27 can you resolve the conflict?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11562][SQL] Provide option to switch Sq...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9553#issuecomment-166025914
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48057/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11562][SQL] Provide option to switch Sq...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9553#issuecomment-165955606
**[Test build #48049 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48049/consoleFull)** for PR 9553 at commit [`8395411`](https://github.com/apache/spark/commit/83954118c6151ec2ae83867d44fa482bac30afd4).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11562][SQL] Provide option to switch Sq...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/9553#discussion_r48095491
--- Diff: repl/scala-2.10/src/main/scala/org/apache/spark/repl/SparkILoop.scala ---
@@ -1026,17 +1027,30 @@ class SparkILoop(
@DeveloperApi
def createSQLContext(): SQLContext = {
- val name = "org.apache.spark.sql.hive.HiveContext"
+ useHiveContext = sparkContext.getConf.getBoolean("spark.sql.useHiveContext", true)
+ val name = {
+ if (useHiveContext) "org.apache.spark.sql.hive.HiveContext"
+ else "org.apache.spark.sql.SQLContext"
+ }
+
val loader = Utils.getContextOrSparkClassLoader
try {
sqlContext = loader.loadClass(name).getConstructor(classOf[SparkContext])
.newInstance(sparkContext).asInstanceOf[SQLContext]
- logInfo("Created sql context (with Hive support)..")
+ if (useHiveContext) {
+ logInfo("Created sql context (with Hive support). To use sqlContext (without Hive), " +
+ "set spark.sql.useHiveContext to false before launching spark-shell.")
+ }
+ else {
+ logInfo("Created sql context.")
+ }
}
catch {
- case _: java.lang.ClassNotFoundException | _: java.lang.NoClassDefFoundError =>
+ case _: java.lang.ClassNotFoundException | _: java.lang.NoClassDefFoundError
+ if useHiveContext =>
sqlContext = new SQLContext(sparkContext)
- logInfo("Created sql context..")
+ logInfo("Created sql context without Hive support, " +
+ "build Spark with -Phive to enable Hive support.")
--- End diff --
`Created sql context without Hive support. To enable Hive support, build Spark with -Phive profile.`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11562][SQL] Provide option to switch Sq...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/9553#discussion_r48095548
--- Diff: repl/scala-2.10/src/main/scala/org/apache/spark/repl/SparkILoop.scala ---
@@ -1026,17 +1027,30 @@ class SparkILoop(
@DeveloperApi
def createSQLContext(): SQLContext = {
- val name = "org.apache.spark.sql.hive.HiveContext"
+ useHiveContext = sparkContext.getConf.getBoolean("spark.sql.useHiveContext", true)
+ val name = {
+ if (useHiveContext) "org.apache.spark.sql.hive.HiveContext"
+ else "org.apache.spark.sql.SQLContext"
+ }
+
val loader = Utils.getContextOrSparkClassLoader
try {
sqlContext = loader.loadClass(name).getConstructor(classOf[SparkContext])
.newInstance(sparkContext).asInstanceOf[SQLContext]
- logInfo("Created sql context (with Hive support)..")
+ if (useHiveContext) {
+ logInfo("Created sql context (with Hive support). To use sqlContext (without Hive), " +
+ "set spark.sql.useHiveContext to false before launching spark-shell.")
+ }
+ else {
+ logInfo("Created sql context.")
+ }
}
catch {
- case _: java.lang.ClassNotFoundException | _: java.lang.NoClassDefFoundError =>
+ case _: java.lang.ClassNotFoundException | _: java.lang.NoClassDefFoundError
+ if useHiveContext =>
sqlContext = new SQLContext(sparkContext)
- logInfo("Created sql context..")
+ logInfo("Created sql context without Hive support, " +
+ "build Spark with -Phive to enable Hive support.")
--- End diff --
It is possible that `-Phive` is used but there was an exception. Can we also update the log message for this case?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11562][SQL] Provide option to switch Sq...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9553#issuecomment-165955621
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11562][SQL] Provide option to switch Sq...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/9553#issuecomment-164592294
I'm not sure if I see the use case. If you don't want a hive context just don't build your jar with `-Phive`? If possible it would be best to not introduce an arbitrary config like the one added here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11562][SQL] Provide option to switch Sq...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9553#issuecomment-166017558
**[Test build #48057 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48057/consoleFull)** for PR 9553 at commit [`969008e`](https://github.com/apache/spark/commit/969008ef85723eb35d4ab038c9158eb3ea12b031).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11562][SQL] Provide option to switch Sq...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9553#issuecomment-165948837
**[Test build #48049 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48049/consoleFull)** for PR 9553 at commit [`8395411`](https://github.com/apache/spark/commit/83954118c6151ec2ae83867d44fa482bac30afd4).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11562][SQL] Provide option to switch Sq...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9553#issuecomment-166025910
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11562][SQL] Provide user an option to i...
Posted by zhzhan <gi...@git.apache.org>.
Github user zhzhan commented on a diff in the pull request:
https://github.com/apache/spark/pull/9553#discussion_r44243045
--- Diff: repl/scala-2.11/src/main/scala/org/apache/spark/repl/Main.scala ---
@@ -78,16 +79,21 @@ object Main extends Logging {
}
sparkContext = new SparkContext(conf)
logInfo("Created spark context..")
+ sqlContextWithHive = conf.getBoolean("spark.sql.hive.context", true)
sparkContext
}
def createSQLContext(): SQLContext = {
- val name = "org.apache.spark.sql.hive.HiveContext"
+ val name = sqlContextWithHive match {
+ case true => "org.apache.spark.sql.hive.HiveContext"
+ case false => "org.apache.spark.sql.SQLContext"
+ }
+
val loader = Utils.getContextOrSparkClassLoader
try {
sqlContext = loader.loadClass(name).getConstructor(classOf[SparkContext])
.newInstance(sparkContext).asInstanceOf[SQLContext]
- logInfo("Created sql context (with Hive support)..")
+ logInfo("Created sql context with " + name)
} catch {
case _: java.lang.ClassNotFoundException | _: java.lang.NoClassDefFoundError =>
sqlContext = new SQLContext(sparkContext)
--- End diff --
I don't know how to fix this, but the logic seems to be weird if SQLContext is used, then in the exception handler the context is created again.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11562][SQL] Provide option to switch Sq...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9553#issuecomment-218706807
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11562][SQL] Provide option to switch Sq...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9553#issuecomment-160813247
**[Test build #46906 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46906/consoleFull)** for PR 9553 at commit [`ce01200`](https://github.com/apache/spark/commit/ce01200604c6feca7cd68918a2f3e15c433827d8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11562][SQL] Provide user an option to i...
Posted by xguo27 <gi...@git.apache.org>.
Github user xguo27 commented on the pull request:
https://github.com/apache/spark/pull/9553#issuecomment-155257787
Hi Zhan:
I just updated documentation and added a guard in the code regarding your feedback on the exception handler.
Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11562][SQL] Provide user an option to i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9553#issuecomment-154902197
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11562][SQL] Provide option to switch Sq...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/9553#issuecomment-165249868
The use case here is that I want to build one common spark that is used across many different clusters. Those clusters may not have Hive running yet, or perhaps I just don't want to have to specify hive-site.xml and include hcatalog, etc. If I'm not using Hive then I don't need spark-shell to load it for me. Perhaps I'm trying to use Spark and they take Hive down to do maintenance, now I can't run Spark or I get error messages out.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11562][SQL] Provide option to switch Sq...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9553#issuecomment-160813342
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46906/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11562][SQL] Provide option to switch Sq...
Posted by xguo27 <gi...@git.apache.org>.
Github user xguo27 commented on the pull request:
https://github.com/apache/spark/pull/9553#issuecomment-165948481
@yhuai I just resolved the conflict. Can we trigger a test? Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11562][SQL] Provide option to switch Sq...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9553#issuecomment-160813341
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11562][SQL] Provide option to switch Sq...
Posted by xguo27 <gi...@git.apache.org>.
Github user xguo27 commented on a diff in the pull request:
https://github.com/apache/spark/pull/9553#discussion_r48095871
--- Diff: repl/scala-2.10/src/main/scala/org/apache/spark/repl/SparkILoop.scala ---
@@ -1026,17 +1027,30 @@ class SparkILoop(
@DeveloperApi
def createSQLContext(): SQLContext = {
- val name = "org.apache.spark.sql.hive.HiveContext"
+ useHiveContext = sparkContext.getConf.getBoolean("spark.sql.useHiveContext", true)
+ val name = {
+ if (useHiveContext) "org.apache.spark.sql.hive.HiveContext"
+ else "org.apache.spark.sql.SQLContext"
+ }
+
val loader = Utils.getContextOrSparkClassLoader
try {
sqlContext = loader.loadClass(name).getConstructor(classOf[SparkContext])
.newInstance(sparkContext).asInstanceOf[SQLContext]
- logInfo("Created sql context (with Hive support)..")
+ if (useHiveContext) {
+ logInfo("Created sql context (with Hive support). To use sqlContext (without Hive), " +
+ "set spark.sql.useHiveContext to false before launching spark-shell.")
+ }
+ else {
+ logInfo("Created sql context.")
+ }
}
catch {
- case _: java.lang.ClassNotFoundException | _: java.lang.NoClassDefFoundError =>
+ case _: java.lang.ClassNotFoundException | _: java.lang.NoClassDefFoundError
+ if useHiveContext =>
sqlContext = new SQLContext(sparkContext)
- logInfo("Created sql context..")
+ logInfo("Created sql context without Hive support, " +
+ "build Spark with -Phive to enable Hive support.")
--- End diff --
When -Phive is used (which provides necessary hive jars) and an exception other than ClassNotFound/NoClassDefFound occured, now how we handle it is to let the exception be propagated without creating an alternative SqlContext.
Do you mean by this case, we should catch -> log -> re-throw?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11562][SQL] Provide option to switch Sq...
Posted by xguo27 <gi...@git.apache.org>.
Github user xguo27 commented on the pull request:
https://github.com/apache/spark/pull/9553#issuecomment-218855824
Yeah sure @andrewor14
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11562][SQL] Provide user an option to i...
Posted by zhzhan <gi...@git.apache.org>.
Github user zhzhan commented on the pull request:
https://github.com/apache/spark/pull/9553#issuecomment-154943558
Need document update for this new configuration.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11562][SQL] Provide user an option to i...
Posted by zhzhan <gi...@git.apache.org>.
Github user zhzhan commented on a diff in the pull request:
https://github.com/apache/spark/pull/9553#discussion_r44242983
--- Diff: repl/scala-2.10/src/main/scala/org/apache/spark/repl/SparkILoop.scala ---
@@ -132,6 +132,7 @@ class SparkILoop(
@DeveloperApi
var sparkContext: SparkContext = _
var sqlContext: SQLContext = _
+ var sqlContextWithHive: Boolean = _
--- End diff --
I don't think we need an extra var for this, and we can get the config directly since nowhere else will use it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11562][SQL] Provide option to switch Sq...
Posted by xguo27 <gi...@git.apache.org>.
Github user xguo27 closed the pull request at:
https://github.com/apache/spark/pull/9553
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org