You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by arman1371 <gi...@git.apache.org> on 2017/01/16 08:41:31 UTC
[GitHub] spark pull request #16598: Added createOrReplaceGlobalTempView method
GitHub user arman1371 opened a pull request:
https://github.com/apache/spark/pull/16598
Added createOrReplaceGlobalTempView method
## What changes were proposed in this pull request?
Added the createOrReplaceGlobalTempView method for dataset
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/arman1371/spark patch-1
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16598.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16598
----
commit a670a3dcff8c14b4cd17f825c949026400ab6f93
Author: Arman <ar...@gmail.com>
Date: 2017-01-16T08:12:54Z
Added createOrReplaceGlobalTempView method
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16598: [SPARK-19236][Core] Added createOrReplaceGlobalTe...
Posted by arman1371 <gi...@git.apache.org>.
Github user arman1371 commented on a diff in the pull request:
https://github.com/apache/spark/pull/16598#discussion_r116350696
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -2603,6 +2603,21 @@ class Dataset[T] private[sql](
def createGlobalTempView(viewName: String): Unit = withPlan {
createTempViewCommand(viewName, replace = false, global = true)
}
+
+ /**
+ * Creates or replaces a global temporary view using the given name. The lifetime of this
+ * temporary view is tied to this Spark application.
+ *
+ * Global temporary view is cross-session. Its lifetime is the lifetime of the Spark application,
+ * i.e. it will be automatically dropped when the application terminates. It's tied to a system
+ * preserved database `_global_temp`, and we must use the qualified name to refer a global temp
+ * view, e.g. `SELECT * FROM _global_temp.view1`.
+ *
+ * @group basic
--- End diff --
But it seems it's not added in 2.1.1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16598: [SPARK-19236][Core] Added createOrReplaceGlobalTempView ...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16598
Merging to master. I will submit a separate PR for resolving the comment. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16598: Added createOrReplaceGlobalTempView method
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16598
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16598: [SPARK-19236][Core] Added createOrReplaceGlobalTe...
Posted by arman1371 <gi...@git.apache.org>.
Github user arman1371 commented on a diff in the pull request:
https://github.com/apache/spark/pull/16598#discussion_r116655733
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -2603,6 +2603,21 @@ class Dataset[T] private[sql](
def createGlobalTempView(viewName: String): Unit = withPlan {
createTempViewCommand(viewName, replace = false, global = true)
}
+
+ /**
+ * Creates or replaces a global temporary view using the given name. The lifetime of this
+ * temporary view is tied to this Spark application.
+ *
+ * Global temporary view is cross-session. Its lifetime is the lifetime of the Spark application,
+ * i.e. it will be automatically dropped when the application terminates. It's tied to a system
+ * preserved database `_global_temp`, and we must use the qualified name to refer a global temp
+ * view, e.g. `SELECT * FROM _global_temp.view1`.
+ *
+ * @group basic
--- End diff --
No, in the API
I edited the comment
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16598: [SPARK-19236][Core] Added createOrReplaceGlobalTempView ...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16598
Sorry, I forgot it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16598: [SPARK-19236][Core] Added createOrReplaceGlobalTe...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16598#discussion_r116674298
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -2603,6 +2603,21 @@ class Dataset[T] private[sql](
def createGlobalTempView(viewName: String): Unit = withPlan {
createTempViewCommand(viewName, replace = false, global = true)
}
+
+ /**
+ * Creates or replaces a global temporary view using the given name. The lifetime of this
+ * temporary view is tied to this Spark application.
+ *
+ * Global temporary view is cross-session. Its lifetime is the lifetime of the Spark application,
+ * i.e. it will be automatically dropped when the application terminates. It's tied to a system
+ * preserved database `_global_temp`, and we must use the qualified name to refer a global temp
+ * view, e.g. `SELECT * FROM _global_temp.view1`.
+ *
+ * @group basic
--- End diff --
This API is added by you in this PR. This is for Scala API. Could you also add the corresponding one to Python API?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16598: [SPARK-19236] Added createOrReplaceGlobalTempView...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/16598#discussion_r96320992
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -2603,6 +2603,21 @@ class Dataset[T] private[sql](
def createGlobalTempView(viewName: String): Unit = withPlan {
createTempViewCommand(viewName, replace = false, global = true)
}
+
+ /**
+ * Creates or replaces a global temporary view using the given name. The lifetime of this
+ * temporary view is tied to this Spark application.
+ *
+ * Global temporary view is cross-session. Its lifetime is the lifetime of the Spark application,
+ * i.e. it will be automatically dropped when the application terminates. It's tied to a system
+ * preserved database `_global_temp`, and we must use the qualified name to refer a global temp
+ * view, e.g. `SELECT * FROM _global_temp.view1`.
+ *
+ * @group basic
--- End diff --
should add since version 2.2.0
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16598: [SPARK-19236][Core] Added createOrReplaceGlobalTempView ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/16598
Hi @arman1371, is it still active?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16598: [SPARK-19236][Core] Added createOrReplaceGlobalTe...
Posted by arman1371 <gi...@git.apache.org>.
Github user arman1371 commented on a diff in the pull request:
https://github.com/apache/spark/pull/16598#discussion_r116410932
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -2603,6 +2603,21 @@ class Dataset[T] private[sql](
def createGlobalTempView(viewName: String): Unit = withPlan {
createTempViewCommand(viewName, replace = false, global = true)
}
+
+ /**
+ * Creates or replaces a global temporary view using the given name. The lifetime of this
+ * temporary view is tied to this Spark application.
+ *
+ * Global temporary view is cross-session. Its lifetime is the lifetime of the Spark application,
+ * i.e. it will be automatically dropped when the application terminates. It's tied to a system
+ * preserved database `_global_temp`, and we must use the qualified name to refer a global temp
+ * view, e.g. `SELECT * FROM _global_temp.view1`.
+ *
+ * @group basic
--- End diff --
The createOrReplaceGlobalTempView method is not in java API
@rxin said it should be added since 2.1.1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16598: [SPARK-19236][Core] Added createOrReplaceGlobalTe...
Posted by arman1371 <gi...@git.apache.org>.
Github user arman1371 commented on a diff in the pull request:
https://github.com/apache/spark/pull/16598#discussion_r118820326
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -2603,6 +2603,21 @@ class Dataset[T] private[sql](
def createGlobalTempView(viewName: String): Unit = withPlan {
createTempViewCommand(viewName, replace = false, global = true)
}
+
+ /**
+ * Creates or replaces a global temporary view using the given name. The lifetime of this
+ * temporary view is tied to this Spark application.
+ *
+ * Global temporary view is cross-session. Its lifetime is the lifetime of the Spark application,
+ * i.e. it will be automatically dropped when the application terminates. It's tied to a system
+ * preserved database `_global_temp`, and we must use the qualified name to refer a global temp
+ * view, e.g. `SELECT * FROM _global_temp.view1`.
+ *
+ * @group basic
--- End diff --
Sorry, I'm not expert in Python
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16598: [SPARK-19236][Core] Added createOrReplaceGlobalTe...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16598#discussion_r116386734
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -2603,6 +2603,21 @@ class Dataset[T] private[sql](
def createGlobalTempView(viewName: String): Unit = withPlan {
createTempViewCommand(viewName, replace = false, global = true)
}
+
+ /**
+ * Creates or replaces a global temporary view using the given name. The lifetime of this
+ * temporary view is tied to this Spark application.
+ *
+ * Global temporary view is cross-session. Its lifetime is the lifetime of the Spark application,
+ * i.e. it will be automatically dropped when the application terminates. It's tied to a system
+ * preserved database `_global_temp`, and we must use the qualified name to refer a global temp
+ * view, e.g. `SELECT * FROM _global_temp.view1`.
+ *
+ * @group basic
--- End diff --
Global temp view support was added in 2.1. See the PR https://github.com/apache/spark/pull/14897
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16598: [SPARK-19236][Core] Added createOrReplaceGlobalTempView ...
Posted by arman1371 <gi...@git.apache.org>.
Github user arman1371 commented on the issue:
https://github.com/apache/spark/pull/16598
@HyukjinKwon Yes
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16598: [SPARK-19236][Core] Added createOrReplaceGlobalTempView ...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/16598
@gatorsmile this didn't run any tests!!!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16598: [SPARK-19236][Core] Added createOrReplaceGlobalTe...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/16598
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16598: [SPARK-19236] Added createOrReplaceGlobalTempView...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/16598#discussion_r96320999
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -2603,6 +2603,21 @@ class Dataset[T] private[sql](
def createGlobalTempView(viewName: String): Unit = withPlan {
createTempViewCommand(viewName, replace = false, global = true)
}
+
+ /**
+ * Creates or replaces a global temporary view using the given name. The lifetime of this
+ * temporary view is tied to this Spark application.
+ *
+ * Global temporary view is cross-session. Its lifetime is the lifetime of the Spark application,
+ * i.e. it will be automatically dropped when the application terminates. It's tied to a system
+ * preserved database `_global_temp`, and we must use the qualified name to refer a global temp
+ * view, e.g. `SELECT * FROM _global_temp.view1`.
+ *
+ * @group basic
--- End diff --
actually 2.1.1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16598: [SPARK-19236][Core] Added createOrReplaceGlobalTe...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16598#discussion_r96361477
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -2603,6 +2603,21 @@ class Dataset[T] private[sql](
def createGlobalTempView(viewName: String): Unit = withPlan {
createTempViewCommand(viewName, replace = false, global = true)
}
+
+ /**
+ * Creates or replaces a global temporary view using the given name. The lifetime of this
+ * temporary view is tied to this Spark application.
+ *
+ * Global temporary view is cross-session. Its lifetime is the lifetime of the Spark application,
+ * i.e. it will be automatically dropped when the application terminates. It's tied to a system
+ * preserved database `_global_temp`, and we must use the qualified name to refer a global temp
+ * view, e.g. `SELECT * FROM _global_temp.view1`.
+ *
+ * @group basic
--- End diff --
We should also port it to Python? https://github.com/apache/spark/blob/master/python/pyspark/sql/dataframe.py
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16598: [SPARK-19236][Core] Added createOrReplaceGlobalTempView ...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the issue:
https://github.com/apache/spark/pull/16598
@gatorsmile, merging this commit ended up breaking tests because this commit fails Scalastyle checks and the blank line actually had trailing whitespace. I'm pushing a hotfix now to address this, but next time please run tests, even for trivial changes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16598: [SPARK-19236][Core] Added createOrReplaceGlobalTe...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16598#discussion_r116625893
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -2603,6 +2603,21 @@ class Dataset[T] private[sql](
def createGlobalTempView(viewName: String): Unit = withPlan {
createTempViewCommand(viewName, replace = false, global = true)
}
+
+ /**
+ * Creates or replaces a global temporary view using the given name. The lifetime of this
+ * temporary view is tied to this Spark application.
+ *
+ * Global temporary view is cross-session. Its lifetime is the lifetime of the Spark application,
+ * i.e. it will be automatically dropped when the application terminates. It's tied to a system
+ * preserved database `_global_temp`, and we must use the qualified name to refer a global temp
+ * view, e.g. `SELECT * FROM _global_temp.view1`.
+ *
+ * @group basic
--- End diff --
JAVA API?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org