You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sh...@apache.org on 2015/10/22 18:34:10 UTC
spark git commit: [SPARK-11244][SPARKR] sparkR.stop() should remove
SQLContext
Repository: spark
Updated Branches:
refs/heads/master c03b6d115 -> 94e2064fa
[SPARK-11244][SPARKR] sparkR.stop() should remove SQLContext
SparkR should remove `.sparkRSQLsc` and `.sparkRHivesc` when `sparkR.stop()` is called. Otherwise even when SparkContext is reinitialized, `sparkRSQL.init` returns the stale copy of the object and complains:
```r
sc <- sparkR.init("local")
sqlContext <- sparkRSQL.init(sc)
sparkR.stop()
sc <- sparkR.init("local")
sqlContext <- sparkRSQL.init(sc)
sqlContext
```
producing
```r
Error in callJMethod(x, "getClass") :
Invalid jobj 1. If SparkR was restarted, Spark operations need to be re-executed.
```
I have added the check and removal only when SparkContext itself is initialized. I have also added corresponding test for this fix. Let me know if you want me to move the test to SQL test suite instead.
p.s. I tried lint-r but ended up a lots of errors on existing code.
Author: Forest Fang <fo...@outlook.com>
Closes #9205 from saurfang/sparkR.stop.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/94e2064f
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/94e2064f
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/94e2064f
Branch: refs/heads/master
Commit: 94e2064fa1b04c05c805d9175c7c78bf583db5c6
Parents: c03b6d1
Author: Forest Fang <fo...@outlook.com>
Authored: Thu Oct 22 09:34:07 2015 -0700
Committer: Shivaram Venkataraman <sh...@cs.berkeley.edu>
Committed: Thu Oct 22 09:34:07 2015 -0700
----------------------------------------------------------------------
R/pkg/R/sparkR.R | 8 ++++++++
R/pkg/inst/tests/test_context.R | 10 ++++++++++
2 files changed, 18 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/94e2064f/R/pkg/R/sparkR.R
----------------------------------------------------------------------
diff --git a/R/pkg/R/sparkR.R b/R/pkg/R/sparkR.R
index 9cf2f1a..043b005 100644
--- a/R/pkg/R/sparkR.R
+++ b/R/pkg/R/sparkR.R
@@ -39,6 +39,14 @@ sparkR.stop <- function() {
sc <- get(".sparkRjsc", envir = env)
callJMethod(sc, "stop")
rm(".sparkRjsc", envir = env)
+
+ if (exists(".sparkRSQLsc", envir = env)) {
+ rm(".sparkRSQLsc", envir = env)
+ }
+
+ if (exists(".sparkRHivesc", envir = env)) {
+ rm(".sparkRHivesc", envir = env)
+ }
}
if (exists(".backendLaunched", envir = env)) {
http://git-wip-us.apache.org/repos/asf/spark/blob/94e2064f/R/pkg/inst/tests/test_context.R
----------------------------------------------------------------------
diff --git a/R/pkg/inst/tests/test_context.R b/R/pkg/inst/tests/test_context.R
index 513bbc8..e99815e 100644
--- a/R/pkg/inst/tests/test_context.R
+++ b/R/pkg/inst/tests/test_context.R
@@ -26,6 +26,16 @@ test_that("repeatedly starting and stopping SparkR", {
}
})
+test_that("repeatedly starting and stopping SparkR SQL", {
+ for (i in 1:4) {
+ sc <- sparkR.init()
+ sqlContext <- sparkRSQL.init(sc)
+ df <- createDataFrame(sqlContext, data.frame(a = 1:20))
+ expect_equal(count(df), 20)
+ sparkR.stop()
+ }
+})
+
test_that("rdd GC across sparkR.stop", {
sparkR.stop()
sc <- sparkR.init() # sc should get id 0
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org