You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by marmbrus <gi...@git.apache.org> on 2014/07/30 00:24:13 UTC
[GitHub] spark pull request: [SPARK-2397][SQL] Deprecate LocalHiveContext
GitHub user marmbrus opened a pull request:
https://github.com/apache/spark/pull/1641
[SPARK-2397][SQL] Deprecate LocalHiveContext
LocalHiveContext is redundant with HiveContext. The only difference is it creates `./metastore` instead of `./metastore_db`.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/marmbrus/spark localHiveContext
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1641.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1641
----
commit 1c2727e47a39e01c6ffb8f4a391d421a51e6f93e
Author: Michael Armbrust <mi...@databricks.com>
Date: 2014-07-29T22:22:42Z
Deprecate LocalHiveContext
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2397][SQL] Deprecate LocalHiveContext
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1641#issuecomment-50703452
QA results for PR 1641:<br>- This patch PASSES unit tests.<br>- This patch merges cleanly<br>- This patch adds the following public classes (experimental):<br>class TestHiveContext(sc: SparkContext) extends HiveContext(sc) {<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17518/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2397][SQL] Deprecate LocalHiveContext
Posted by nightwolfzor <gi...@git.apache.org>.
Github user nightwolfzor commented on the pull request:
https://github.com/apache/spark/pull/1641#issuecomment-63597374
Just curious why this has been Deprecated. It was rather useful for testing purposes when you run to run an "embedded" hive instance. And not supply a hive site. Would you now advise for tests that a Hive context be created but we go the extra step and manually set the conf like `LocalHiveContext` did?
I.e. something like
`setConf("javax.jdo.option.ConnectionURL",
s"jdbc:derby:;databaseName=$metastorePath;create=true")
setConf("hive.metastore.warehouse.dir", warehousePath)`
Or am I doing something stupid?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2397][SQL] Deprecate LocalHiveContext
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/1641
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2397][SQL] Deprecate LocalHiveContext
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1641#issuecomment-50548333
QA tests have started for PR 1641. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17387/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2397][SQL] Deprecate LocalHiveContext
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1641#issuecomment-50554626
QA results for PR 1641:<br>- This patch FAILED unit tests.<br>- This patch merges cleanly<br>- This patch adds the following public classes (experimental):<br>class TestHiveContext(sc: SparkContext) extends HiveContext(sc) {<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17387/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2397][SQL] Deprecate LocalHiveContext
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1641#issuecomment-50698840
QA tests have started for PR 1641. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17518/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2397][SQL] Deprecate LocalHiveContext
Posted by ahirreddy <gi...@git.apache.org>.
Github user ahirreddy commented on a diff in the pull request:
https://github.com/apache/spark/pull/1641#discussion_r15623100
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala ---
@@ -42,9 +42,12 @@ import org.apache.spark.sql.execution.{Command => PhysicalCommand}
import org.apache.spark.sql.hive.execution.DescribeHiveTableCommand
/**
- * Starts up an instance of hive where metadata is stored locally. An in-process metadata data is
- * created with data stored in ./metadata. Warehouse data is stored in in ./warehouse.
+ * DEPRECATED: Use HiveContext instead.
*/
+@deprecated("""
--- End diff --
Ironically, I think ```@deprecated``` with 1 parameter is deprecated. The 2 parameter version that takes message and version number should do the trick.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2397][SQL] Deprecate LocalHiveContext
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/1641#discussion_r15624618
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala ---
@@ -42,9 +42,12 @@ import org.apache.spark.sql.execution.{Command => PhysicalCommand}
import org.apache.spark.sql.hive.execution.DescribeHiveTableCommand
/**
- * Starts up an instance of hive where metadata is stored locally. An in-process metadata data is
- * created with data stored in ./metadata. Warehouse data is stored in in ./warehouse.
+ * DEPRECATED: Use HiveContext instead.
*/
+@deprecated("""
--- End diff --
Lol... fixed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2397][SQL] Deprecate LocalHiveContext
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/1641#discussion_r15624582
--- Diff: python/pyspark/sql.py ---
@@ -813,6 +815,10 @@ class LocalHiveContext(HiveContext):
130091
"""
+ def __init__(self, sparkContext, sqlContext=None):
+ HiveContext.__init__(self, sparkContext, sqlContext)
+ warnings.warn("LocalHiveContext is deprecated. Use HiveContext instead.", DeprecationWarning)
--- End diff --
@JoshRosen informs me that you have to turn on warnings to see it. Perhaps we should consider doing that for pyspark.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2397][SQL] Deprecate LocalHiveContext
Posted by ahirreddy <gi...@git.apache.org>.
Github user ahirreddy commented on a diff in the pull request:
https://github.com/apache/spark/pull/1641#discussion_r15623130
--- Diff: python/pyspark/sql.py ---
@@ -813,6 +815,10 @@ class LocalHiveContext(HiveContext):
130091
"""
+ def __init__(self, sparkContext, sqlContext=None):
+ HiveContext.__init__(self, sparkContext, sqlContext)
+ warnings.warn("LocalHiveContext is deprecated. Use HiveContext instead.", DeprecationWarning)
--- End diff --
Did we find out if this ever prints anywhere?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2397][SQL] Deprecate LocalHiveContext
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/1641#issuecomment-50798877
Thanks for the review! Merged into master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2397][SQL] Deprecate LocalHiveContext
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/1641#issuecomment-63690052
Creating a `HiveContext` without a `hive-site.xml` will also work, and will put the warehouse/metastore_db in the same place as Hive/Shark would. This is why we deprecated the `LocalHiveContext`.
Also for testing I would consider using `TestHive` (which is a mostly stable developer api) as it creates a fresh metastore each time it is started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org