You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by gatorsmile <gi...@git.apache.org> on 2017/05/30 21:26:05 UTC
[GitHub] spark pull request #18147: [SPARK-19236][CORE] Added createOrReplaceGlobalTe...
GitHub user gatorsmile opened a pull request:
https://github.com/apache/spark/pull/18147
[SPARK-19236][CORE] Added createOrReplaceGlobalTempView method
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
Please review http://spark.apache.org/contributing.html before opening a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gatorsmile/spark createOrReplaceGlobalTempView
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18147.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18147
----
commit e266bee680b6f565e0e80655947d37666ad806fd
Author: gatorsmile <ga...@gmail.com>
Date: 2017-05-30T21:23:29Z
fix
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18147: [SPARK-19236][SQL][FOLLOW-UP] Added createOrRepla...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/18147
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18147: [SPARK-19236][SQL][FOLLOW-UP] Added createOrReplaceGloba...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18147
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18147: [SPARK-19236][SQL][FOLLOW-UP] Added createOrReplaceGloba...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18147
**[Test build #77563 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77563/testReport)** for PR 18147 at commit [`9191609`](https://github.com/apache/spark/commit/91916099b52b217fb676353719fabd735052fddd).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18147: [SPARK-19236][SQL][FOLLOW-UP] Added createOrRepla...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/18147#discussion_r119267756
--- Diff: python/pyspark/sql/dataframe.py ---
@@ -191,6 +191,23 @@ def createGlobalTempView(self, name):
"""
self._jdf.createGlobalTempView(name)
+ @since(2.2)
--- End diff --
we didn't merge the original PR to 2.2 right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18147: [SPARK-19236][SQL][FOLLOW-UP] Added createOrReplaceGloba...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/18147
cc @cloud-fan @ueshin
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18147: [SPARK-19236][SQL][FOLLOW-UP] Added createOrReplaceGloba...
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18147
LGTM except for @cloud-fan's comment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18147: [SPARK-19236][SQL][FOLLOW-UP] Added createOrRepla...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/18147#discussion_r119222688
--- Diff: python/pyspark/sql/dataframe.py ---
@@ -191,6 +191,23 @@ def createGlobalTempView(self, name):
"""
self._jdf.createGlobalTempView(name)
+ @since(2.2)
+ def createOrReplaceGlobalTempView(self, name):
+ """Creates or replaces a global temporary view using the given name.
+
+ The lifetime of this temporary view is tied to this Spark application.
+
+ >>> df.createOrReplaceGlobalTempView("people")
+ >>> df2 = df.filter(df.age > 3)
+ >>> df2.createOrReplaceGlobalTempView("people")
+ >>> df3 = spark.sql("select * from people")
--- End diff --
`df3 = spark.sql("select * from _global_temp.people")`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18147: [SPARK-19236][SQL][FOLLOW-UP] Added createOrReplaceGloba...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18147
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77563/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18147: [SPARK-19236][SQL][FOLLOW-UP] Added createOrReplaceGloba...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18147
thanks, merging to master!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18147: [SPARK-19236][SQL][FOLLOW-UP] Added createOrReplaceGloba...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18147
**[Test build #77552 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77552/testReport)** for PR 18147 at commit [`e266bee`](https://github.com/apache/spark/commit/e266bee680b6f565e0e80655947d37666ad806fd).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18147: [SPARK-19236][SQL][FOLLOW-UP] Added createOrReplaceGloba...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18147
**[Test build #77552 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77552/testReport)** for PR 18147 at commit [`e266bee`](https://github.com/apache/spark/commit/e266bee680b6f565e0e80655947d37666ad806fd).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18147: [SPARK-19236][SQL][FOLLOW-UP] Added createOrRepla...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/18147#discussion_r119436898
--- Diff: python/pyspark/sql/dataframe.py ---
@@ -191,6 +191,23 @@ def createGlobalTempView(self, name):
"""
self._jdf.createGlobalTempView(name)
+ @since(2.2)
--- End diff --
See the backport PR https://github.com/apache/spark/pull/18167
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18147: [SPARK-19236][SQL][FOLLOW-UP] Added createOrReplaceGloba...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18147
**[Test build #77563 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77563/testReport)** for PR 18147 at commit [`9191609`](https://github.com/apache/spark/commit/91916099b52b217fb676353719fabd735052fddd).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18147: [SPARK-19236][SQL][FOLLOW-UP] Added createOrReplaceGloba...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18147
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18147: [SPARK-19236][SQL][FOLLOW-UP] Added createOrRepla...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/18147#discussion_r119246631
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/GlobalTempViewSuite.scala ---
@@ -35,39 +35,47 @@ class GlobalTempViewSuite extends QueryTest with SharedSQLContext {
private var globalTempDB: String = _
test("basic semantic") {
- sql("CREATE GLOBAL TEMP VIEW src AS SELECT 1, 'a'")
+ try {
+ sql("CREATE GLOBAL TEMP VIEW src AS SELECT 1, 'a'")
+
+ // If there is no database in table name, we should try local temp view first, if not found,
+ // try table/view in current database, which is "default" in this case. So we expect
+ // NoSuchTableException here.
+ intercept[NoSuchTableException](spark.table("src"))
- // If there is no database in table name, we should try local temp view first, if not found,
- // try table/view in current database, which is "default" in this case. So we expect
- // NoSuchTableException here.
- intercept[NoSuchTableException](spark.table("src"))
+ // Use qualified name to refer to the global temp view explicitly.
+ checkAnswer(spark.table(s"$globalTempDB.src"), Row(1, "a"))
- // Use qualified name to refer to the global temp view explicitly.
- checkAnswer(spark.table(s"$globalTempDB.src"), Row(1, "a"))
+ // Table name without database will never refer to a global temp view.
+ intercept[NoSuchTableException](sql("DROP VIEW src"))
- // Table name without database will never refer to a global temp view.
- intercept[NoSuchTableException](sql("DROP VIEW src"))
+ sql(s"DROP VIEW $globalTempDB.src")
+ // The global temp view should be dropped successfully.
+ intercept[NoSuchTableException](spark.table(s"$globalTempDB.src"))
- sql(s"DROP VIEW $globalTempDB.src")
- // The global temp view should be dropped successfully.
- intercept[NoSuchTableException](spark.table(s"$globalTempDB.src"))
+ // We can also use Dataset API to create global temp view
+ Seq(1 -> "a").toDF("i", "j").createGlobalTempView("src")
+ checkAnswer(spark.table(s"$globalTempDB.src"), Row(1, "a"))
- // We can also use Dataset API to create global temp view
- Seq(1 -> "a").toDF("i", "j").createGlobalTempView("src")
- checkAnswer(spark.table(s"$globalTempDB.src"), Row(1, "a"))
+ // Use qualified name to rename a global temp view.
+ sql(s"ALTER VIEW $globalTempDB.src RENAME TO src2")
+ intercept[NoSuchTableException](spark.table(s"$globalTempDB.src"))
+ checkAnswer(spark.table(s"$globalTempDB.src2"), Row(1, "a"))
- // Use qualified name to rename a global temp view.
- sql(s"ALTER VIEW $globalTempDB.src RENAME TO src2")
- intercept[NoSuchTableException](spark.table(s"$globalTempDB.src"))
- checkAnswer(spark.table(s"$globalTempDB.src2"), Row(1, "a"))
+ // Use qualified name to alter a global temp view.
+ sql(s"ALTER VIEW $globalTempDB.src2 AS SELECT 2, 'b'")
+ checkAnswer(spark.table(s"$globalTempDB.src2"), Row(2, "b"))
- // Use qualified name to alter a global temp view.
- sql(s"ALTER VIEW $globalTempDB.src2 AS SELECT 2, 'b'")
- checkAnswer(spark.table(s"$globalTempDB.src2"), Row(2, "b"))
+ // We can also use Catalog API to drop global temp view
+ spark.catalog.dropGlobalTempView("src2")
+ intercept[NoSuchTableException](spark.table(s"$globalTempDB.src2"))
- // We can also use Catalog API to drop global temp view
- spark.catalog.dropGlobalTempView("src2")
- intercept[NoSuchTableException](spark.table(s"$globalTempDB.src2"))
+ // We can also use Dataset API to replace global temp view
--- End diff --
The code added are line 38 and line 73-78
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18147: [SPARK-19236][SQL][FOLLOW-UP] Added createOrReplaceGloba...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18147
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77552/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org