You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by dongjoon-hyun <gi...@git.apache.org> on 2017/05/10 22:53:44 UTC
[GitHub] spark pull request #17941: [SPARK-20684][R] Expose createGlobalTempView and ...
GitHub user dongjoon-hyun opened a pull request:
https://github.com/apache/spark/pull/17941
[SPARK-20684][R] Expose createGlobalTempView and dropGlobalTempView in SparkR
## What changes were proposed in this pull request?
This PR aims to expose `createGlobalTempView` and `dropGlobalTempView` in SparkR.
## How was this patch tested?
Pass the Jenkins.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dongjoon-hyun/spark SPARK-20684
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17941.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17941
----
commit ec3957674110c21ef6d065be083f2052818b6450
Author: Dongjoon Hyun <do...@apache.org>
Date: 2017-05-10T22:50:21Z
[SPARK-20684][R] Expose createGlobalTempView and dropGlobalTempView in SparkR
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose createGlobalTempView and dropGlo...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/17941
Yep. It looks like that. At that time, @rxin was the reporter. And this time, @falaki is the reporter.
@felixcheung . Zeppelin can share the Spark context between Java and Scala, can't it?
Can Databricks cloud share the context (or JVM) among more languages? If then, that would be great.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose createGlobalTempView and dropGlo...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/17941
Oh, I assumed there exists a case about cross languages inside SparkR and thought this is another API parity issue at first. Maybe, @falaki can give us the goal described in JIRA.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose createGlobalTempView and dropGlo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17941
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17941: [SPARK-20684][R] Expose createGlobalTempView and ...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/17941#discussion_r115882801
--- Diff: R/pkg/R/DataFrame.R ---
@@ -501,6 +501,34 @@ setMethod("createOrReplaceTempView",
invisible(callJMethod(x@sdf, "createOrReplaceTempView", viewName))
})
+#' Creates a global temporary view using the given name.
--- End diff --
I updated the doc.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose createGlobalTempView and dropGlo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17941
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose createGlobalTempView and dropGlo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17941
**[Test build #77211 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77211/testReport)** for PR 17941 at commit [`2e43c14`](https://github.com/apache/spark/commit/2e43c147468904633b3c372637af93e9e2282799).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose create[OrReplace]GlobalTempView ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17941
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose create[OrReplace]GlobalTempView ...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/17941
Could you review this, @felixcheung ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17941: [SPARK-20684][R] Expose create[OrReplace]GlobalTe...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/17941#discussion_r121255138
--- Diff: R/pkg/R/DataFrame.R ---
@@ -501,6 +501,72 @@ setMethod("createOrReplaceTempView",
invisible(callJMethod(x@sdf, "createOrReplaceTempView", viewName))
})
+#' Creates a global temporary view using the given name.
+#'
+#' Creates a new global temporary view using a SparkDataFrame in the Spark Session.
+#' Global temporary view is cross-session while a temporary view is not. Its lifetime
+#' is the lifetime of the Spark application, i.e. it will be automatically dropped
+#' when the application terminates. It's tied to a system preserved database `global_temp`,
+#' and we must use the qualified name to refer a global temporary view, e.g.
+#' `SELECT * FROM global_temp.view1`. If a global temporary view with the same name
+#' already exists, it raises exception.
--- End diff --
I guess `` ` ... ` `` should be `\code{ ... }` if this is meant to be a code block.
BTW, I hope the documentations for the same API in Scala/Python/R are almost identical if these are not language-specific. This requires a bit of overhead when someone sweep them to fix the same contents and sometimes it needs several follow-up PRs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose createGlobalTempView and dropGlo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17941
**[Test build #77649 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77649/testReport)** for PR 17941 at commit [`2e43c14`](https://github.com/apache/spark/commit/2e43c147468904633b3c372637af93e9e2282799).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17941: [SPARK-20684][R] Expose createGlobalTempView and ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/17941#discussion_r119700913
--- Diff: R/pkg/NAMESPACE ---
@@ -98,6 +98,7 @@ exportMethods("arrange",
"corr",
"covar_samp",
"covar_pop",
+ "createGlobalTempView",
--- End diff --
there is also a `createOrReplaceGlobalTempView`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose createGlobalTempView and dropGlo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17941
**[Test build #76760 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76760/testReport)** for PR 17941 at commit [`2e43c14`](https://github.com/apache/spark/commit/2e43c147468904633b3c372637af93e9e2282799).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17941: [SPARK-20684][R] Expose createGlobalTempView and ...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/17941#discussion_r115882951
--- Diff: R/pkg/inst/tests/testthat/test_sparkSQL.R ---
@@ -721,6 +721,17 @@ test_that(
expect_true(dropTempView("dfView"))
})
+test_that(
+ "createGlobalTempView() results in a queryable table and sql() results in a new DataFrame", {
+ df <- read.json(jsonPath)
+ createGlobalTempView(df, "global_table1")
+ newdf <- sql("SELECT * FROM global_temp.global_table1 where name = 'Michael'")
+ expect_is(newdf, "SparkDataFrame")
+ expect_equal(count(newdf), 1)
+ expect_true(dropGlobalTempView("global_table1"))
+ expect_false(dropGlobalTempView("global_table1"))
--- End diff --
Instead, I added the above line to check the success of deletion in the previous line. How do you think about this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose create[OrReplace]GlobalTempView ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17941
**[Test build #77653 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77653/testReport)** for PR 17941 at commit [`6dd88e2`](https://github.com/apache/spark/commit/6dd88e239e6aca3a117461d2f34bc6f917e29f8f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose createGlobalTempView and dropGlo...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on the issue:
https://github.com/apache/spark/pull/17941
@felixcheung How about before `sparkR.session.stop()`? The use case I am trying to recover (which is broken now) is sharing table in SparkR and working on it in a different language.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose createGlobalTempView and dropGlo...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/17941
actually, it was intentional that we didn't add it in R since there's always one session only.
Could you elaborate how global view helps with cross language?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose createGlobalTempView and dropGlo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17941
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17941: [SPARK-20684][R] Expose createGlobalTempView and ...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/17941#discussion_r119705853
--- Diff: R/pkg/NAMESPACE ---
@@ -98,6 +98,7 @@ exportMethods("arrange",
"corr",
"covar_samp",
"covar_pop",
+ "createGlobalTempView",
--- End diff --
Oh, I'll expose that too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose createGlobalTempView and dropGlo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17941
**[Test build #77211 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77211/testReport)** for PR 17941 at commit [`2e43c14`](https://github.com/apache/spark/commit/2e43c147468904633b3c372637af93e9e2282799).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose createGlobalTempView and dropGlo...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/17941
Thank you for comments, @falaki and @felixcheung . I added the duplication link to the issue, SPARK-20684, and ask @falaki to close the JIRA issue because he is the reporter.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose createGlobalTempView and dropGlo...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/17941
Thank you for confirming. If then, what about merging this for both API parity in Apache Spark and preparing the furture?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose createGlobalTempView and dropGlo...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on the issue:
https://github.com/apache/spark/pull/17941
@felixcheung we all know that SparkR (and in general R) API is not perfect when it comes to ETLing unstructured data. For example we don't have a great story for nested data, etc. To overcome these limitations many ETL their data in Python or Scala and then analyze them in R.
With introduction of sessions that workflow is partially broken. You can still do it but you need to persist the table. The global temp view is to solve that problem. It exists in PySpark, so I think it deserves to exist in SparkR as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17941: [SPARK-20684][R] Expose createGlobalTempView and ...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/17941#discussion_r115879381
--- Diff: R/pkg/inst/tests/testthat/test_sparkSQL.R ---
@@ -721,6 +721,16 @@ test_that(
expect_true(dropTempView("dfView"))
})
+test_that(
+ "createGlobalTempView() results in a queryable table and sql() results in a new DataFrame", {
+ df <- read.json(jsonPath)
+ createGlobalTempView(df, "global_table1")
+ newdf <- sql("SELECT * FROM global_temp.global_table1 where name = 'Michael'")
+ expect_is(newdf, "SparkDataFrame")
+ expect_equal(count(newdf), 1)
+ expect_true(dropGlobalTempView("global_table1"))
--- End diff --
Ur, if you don't mind, I will use the way to count tables like
https://github.com/apache/spark/commit/b952b44af4d243f1e3ad88bccf4af7d04df3fc81 .
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose createGlobalTempView and dropGlo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17941
**[Test build #76760 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76760/testReport)** for PR 17941 at commit [`2e43c14`](https://github.com/apache/spark/commit/2e43c147468904633b3c372637af93e9e2282799).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose create[OrReplace]GlobalTempView ...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/17941
Hyukjin appreciate your chiming in.
This is something that does not work for R (in SparkR standalone) specifically. For reasons and proposal to address please see the earlier comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose createGlobalTempView and dropGlo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17941
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77649/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose createGlobalTempView and dropGlo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17941
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose createGlobalTempView and dropGlo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17941
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76760/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose createGlobalTempView and dropGlo...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/17941
there are several different ways to share context or `SparkSession` between languages and I think that discussion is orthogonal to the global session.
is there value having temp view in global session? do temp views in global session disappear when the SparkContext is stopped? that was my understanding earlier and I offered in SPARK-17865 several different approaches where the global session would make sense in that case. otherwise since SparkR itself shutdown both SparkSession and SparkContext together, I think it will be very confusing to introduce an API that "does not work" in the SparkR package.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose create[OrReplace]GlobalTempView ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/17941
Ah, so, this API is useless in R alone but might be useful when it shares in other languages for the current status. The point is that this API has to be useful in R alone as well (and this should be primary) rather than accepting the current status. I see.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17941: [SPARK-20684][R] Expose create[OrReplace]GlobalTe...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun closed the pull request at:
https://github.com/apache/spark/pull/17941
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose createGlobalTempView and dropGlo...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/17941
Also, ping @shivaram and @yanboliang .
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose create[OrReplace]GlobalTempView ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17941
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77653/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose createGlobalTempView and dropGlo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17941
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose create[OrReplace]GlobalTempView ...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/17941
If @felixcheung provides the real open source implementation for that in a near future, that would be perfect. Why don't you make a PR as a PoC first if the implementation isn't going to be hard?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose createGlobalTempView and dropGlo...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/17941
@felixcheung what's your concern with this one? seems like just for api parity sake we should add this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose createGlobalTempView and dropGlo...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/17941
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose createGlobalTempView and dropGlo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17941
**[Test build #77650 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77650/testReport)** for PR 17941 at commit [`74b4453`](https://github.com/apache/spark/commit/74b445375db00daabcaaa3513a1dc66797984bcd).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17941: [SPARK-20684][R] Expose createGlobalTempView and ...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/17941#discussion_r115872802
--- Diff: R/pkg/R/DataFrame.R ---
@@ -501,6 +501,34 @@ setMethod("createOrReplaceTempView",
invisible(callJMethod(x@sdf, "createOrReplaceTempView", viewName))
})
+#' Creates a global temporary view using the given name.
--- End diff --
Would you please extend this documentation to explain what is the difference between a global temp view and a normal temp view? I constantly hear from SparkR users that our documentation simply lists the function arguments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose createGlobalTempView and dropGlo...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/17941
@rxin it was from a while back in SPARK-17865
essentially these are exactly the same in standalone SparkR:
```
sparkR.session()
...
createGlobalTempView(df, "foo")
sparkR.session.stop()
```
same as
```
sparkR.session()
...
createOrReplaceTempView(df, "foo")
sparkR.session.stop()
```
this is because in SparkR, `SparkContext` is stopped the same time as `SparkSession` (and there is only one session at a time)
so I think it makes sense to formally support the idea of the "global session" as discussed in SPARK-17865; with some changes in session initialization and session termination, and R session cleanup, plus more documentation and proper testing of global view lifetime beyond session, it should be good to have.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose create[OrReplace]GlobalTempView ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/17941
Just a question out of curiosity. Don't we have this in Python too? I just wonder if this support should be in a different way and/or held off for R only. Is there any case it does not work incorrectly R-specifically?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose createGlobalTempView and dropGlo...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/17941
Retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose createGlobalTempView and dropGlo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17941
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77211/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose createGlobalTempView and dropGlo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17941
**[Test build #77649 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77649/testReport)** for PR 17941 at commit [`2e43c14`](https://github.com/apache/spark/commit/2e43c147468904633b3c372637af93e9e2282799).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose create[OrReplace]GlobalTempView ...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/17941
Hi, @felixcheung .
IMO, this is just a confirmation on interface. This will only help future contributors (including you) make a PR on the implementation.
Or, if you want to include this PR in your PR, please make your PR first. I will take a look at that and close this happily.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose createGlobalTempView and dropGlo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17941
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77650/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose createGlobalTempView and dropGlo...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on the issue:
https://github.com/apache/spark/pull/17941
@dongjoon-hyun and @felixcheung just to confirm, yes we do share SparkContext across languages in Databricks.
I think this is a useful API in general. Note that when user creates spark session in SparkR if there is an existing `SQLContext` that is returned.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose createGlobalTempView and dropGlo...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/17941
Given the timing are you suggesting we include this in 2.2.0?
This hasn't been review yet.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17941: [SPARK-20684][R] Expose createGlobalTempView and ...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/17941#discussion_r115881192
--- Diff: R/pkg/inst/tests/testthat/test_sparkSQL.R ---
@@ -721,6 +721,16 @@ test_that(
expect_true(dropTempView("dfView"))
})
+test_that(
+ "createGlobalTempView() results in a queryable table and sql() results in a new DataFrame", {
+ df <- read.json(jsonPath)
+ createGlobalTempView(df, "global_table1")
+ newdf <- sql("SELECT * FROM global_temp.global_table1 where name = 'Michael'")
+ expect_is(newdf, "SparkDataFrame")
+ expect_equal(count(newdf), 1)
+ expect_true(dropGlobalTempView("global_table1"))
--- End diff --
I found that global temp tables are now returned by `listTables`. (R is the same).
```
scala> spark.range(10).createGlobalTempView("t1")
scala> spark.catalog.tableExists("global_temp.t1")
res8: Boolean = true
scala> spark.catalog.listTables().show
+----+--------+-----------+---------+-----------+
|name|database|description|tableType|isTemporary|
+----+--------+-----------+---------+-----------+
+----+--------+-----------+---------+-----------+
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose createGlobalTempView and dropGlo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17941
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76759/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose create[OrReplace]GlobalTempView ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17941
**[Test build #77653 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77653/testReport)** for PR 17941 at commit [`6dd88e2`](https://github.com/apache/spark/commit/6dd88e239e6aca3a117461d2f34bc6f917e29f8f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17941: [SPARK-20684][R] Expose createGlobalTempView and ...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/17941#discussion_r115877250
--- Diff: R/pkg/inst/tests/testthat/test_sparkSQL.R ---
@@ -721,6 +721,16 @@ test_that(
expect_true(dropTempView("dfView"))
})
+test_that(
+ "createGlobalTempView() results in a queryable table and sql() results in a new DataFrame", {
+ df <- read.json(jsonPath)
+ createGlobalTempView(df, "global_table1")
+ newdf <- sql("SELECT * FROM global_temp.global_table1 where name = 'Michael'")
+ expect_is(newdf, "SparkDataFrame")
+ expect_equal(count(newdf), 1)
+ expect_true(dropGlobalTempView("global_table1"))
--- End diff --
Sure!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose createGlobalTempView and dropGlo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17941
**[Test build #76759 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76759/testReport)** for PR 17941 at commit [`ec39576`](https://github.com/apache/spark/commit/ec3957674110c21ef6d065be083f2052818b6450).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose create[OrReplace]GlobalTempView ...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/17941
@cloud-fan thank you for chiming in. I have been looking for some feedback and I'm hoping we could get something more definitive.
I'm sure there is a need to share context between different languages. Having been on the consuming side for a long time I'm surprised the lack of incentive to formalize how to share context, and to avoid private methods getting called, wire protocol reverse-engineered and so on.
I'm aware of at least 5 or 6 unique implementations of `R - to - Spark`. Among these, I'm pretty certain only `databricks` has the proper support for multiple SparkSessions. There is only 1 other implementation that has sharing and it does so at the SparkSession-level (ie. there is only one session across all languages).
So I think sharing context is an orthogonal question, and can be achieved in several different ways.
Even if there isn't multiple/default/active session support, or that it wouldn't applicable, I still think it makes sense to have the concept of a global session, if nothing else we could get that API parity. So all I'm saying is we should have that in SparkR so that any other APIs working with global sessions only would actually work with SparkR by itself.
And for that, a proposal was made, and from what I can see implementation isn't going to be hard.
What do people think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose createGlobalTempView and dropGlo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17941
**[Test build #77650 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77650/testReport)** for PR 17941 at commit [`74b4453`](https://github.com/apache/spark/commit/74b445375db00daabcaaa3513a1dc66797984bcd).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose createGlobalTempView and dropGlo...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/17941
FYI, this was SPARK-17865 and decided not needed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17941: [SPARK-20684][R] Expose createGlobalTempView and ...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/17941#discussion_r115873162
--- Diff: R/pkg/inst/tests/testthat/test_sparkSQL.R ---
@@ -721,6 +721,16 @@ test_that(
expect_true(dropTempView("dfView"))
})
+test_that(
+ "createGlobalTempView() results in a queryable table and sql() results in a new DataFrame", {
+ df <- read.json(jsonPath)
+ createGlobalTempView(df, "global_table1")
+ newdf <- sql("SELECT * FROM global_temp.global_table1 where name = 'Michael'")
+ expect_is(newdf, "SparkDataFrame")
+ expect_equal(count(newdf), 1)
+ expect_true(dropGlobalTempView("global_table1"))
--- End diff --
Please verify that the view is indeed dropped by comparing result of `tableNames()` after the drop call to what it was at the beginning.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose createGlobalTempView and dropGlo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17941
**[Test build #76759 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76759/testReport)** for PR 17941 at commit [`ec39576`](https://github.com/apache/spark/commit/ec3957674110c21ef6d065be083f2052818b6450).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17941: [SPARK-20684][R] Expose createGlobalTempView and dropGlo...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/17941
`sparkR.session.stop()` is a problem, but I think this patch still makes sense to share global temp views between different sessions, e.g. between Spark R and PySpark. We should not block this PR because of it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org