You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by shivaram <gi...@git.apache.org> on 2016/08/23 19:49:14 UTC
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
GitHub user shivaram opened a pull request:
https://github.com/apache/spark/pull/14775
[SPARK-16581][SPARKR] Make JVM backend calling functions public
## What changes were proposed in this pull request?
This change exposes a public API in SparkR to create objects, call methods on the Spark driver JVM
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
Unit tests, CRAN checks
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/shivaram/spark-1 sparkr-java-api
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/14775.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #14775
----
commit 472072ae12c146bd1a7a844337c4496bbdfdbd2d
Author: Shivaram Venkataraman <sh...@cs.berkeley.edu>
Date: 2016-08-22T19:37:38Z
First cut of making JVM functions public
commit d772285ab1d6ed7facd5db05b772ba752428a30f
Author: Shivaram Venkataraman <sh...@cs.berkeley.edu>
Date: 2016-08-23T17:29:43Z
Improve documentation for JVM methods
commit 5f4b53ad03f6c6b5d4d643a8e6161856b0452704
Author: Shivaram Venkataraman <sh...@cs.berkeley.edu>
Date: 2016-08-23T17:29:59Z
Merge branch 'master' of https://github.com/apache/spark into sparkr-java-api
commit 5f2936165e7c9fa00bd589563a6b0a324b1e7989
Author: Shivaram Venkataraman <sh...@cs.berkeley.edu>
Date: 2016-08-23T18:30:52Z
Merge branch 'master' of https://github.com/apache/spark into sparkr-java-api
commit d267f2f271aa4e3f7bd8542fa1772ecc978eeee8
Author: Shivaram Venkataraman <sh...@cs.berkeley.edu>
Date: 2016-08-23T19:40:39Z
Add unit tests for JVM API
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r76522953
--- Diff: R/pkg/R/backend.R ---
@@ -37,12 +51,42 @@ callJMethod <- function(objId, methodName, ...) {
invokeJava(isStatic = FALSE, objId$id, methodName, ...)
}
-# Call a static method on a specified className
+#' Call Static Java Methods
+#'
+#' Call a static method in the JVM running the Spark driver.
+#'
+#' @param className class containing the static method to invoke.
--- End diff --
Done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r76523010
--- Diff: R/pkg/R/jobj.R ---
@@ -82,7 +82,20 @@ getClassName.jobj <- function(x) {
callJMethod(cls, "getName")
}
-cleanup.jobj <- function(jobj) {
+#' Garbage collect Java Objects
+#'
+#' Garbage collect an object allocated on Spark driver JVM heap.
+#'
+#' cleanup.jobj is a low level method that lets developers manually garbage collect objects
+#' allocated using newJObject. This is only to be used for advanced use cases as objects allocated
+#' on the JVM heap are automatically garbage collected when the corresponding R reference goes out
+#' of scope.
+#'
+#' @param x the Java object that should be garbage collected.
+#' @note cleanup.jobj since 2.0.1
+#' @export
+cleanup.jobj <- function(x) {
+ jobj <- x
if (isValidJobj(jobj)) {
--- End diff --
I guess since this is back to an internal method we dont need to worry about it. But I guess I can add the check if we still want it ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r76522995
--- Diff: R/pkg/R/jobj.R ---
@@ -82,7 +82,20 @@ getClassName.jobj <- function(x) {
callJMethod(cls, "getName")
}
-cleanup.jobj <- function(jobj) {
+#' Garbage collect Java Objects
+#'
+#' Garbage collect an object allocated on Spark driver JVM heap.
+#'
+#' cleanup.jobj is a low level method that lets developers manually garbage collect objects
+#' allocated using newJObject. This is only to be used for advanced use cases as objects allocated
+#' on the JVM heap are automatically garbage collected when the corresponding R reference goes out
+#' of scope.
+#'
+#' @param x the Java object that should be garbage collected.
+#' @note cleanup.jobj since 2.0.1
+#' @export
+cleanup.jobj <- function(x) {
--- End diff --
Yeah thats a good point - but I've removed this as we don't need to export this for now
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r75956122
--- Diff: R/pkg/R/backend.R ---
@@ -37,12 +51,42 @@ callJMethod <- function(objId, methodName, ...) {
invokeJava(isStatic = FALSE, objId$id, methodName, ...)
}
-# Call a static method on a specified className
+#' Call Static Java Methods
+#'
+#' Call a static method in the JVM running the Spark driver.
+#'
+#' @param className class containing the static method to invoke.
+#' @param methodName name of static method to invoke.
+#' @param ... parameters to pass to the Java method.
+#' @export
+#' @seealso callJMethod, newJObject
+#' @examples
+#' \dontrun{
--- End diff --
@aliases, @note since 2.0.1?
@return
@rdname
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r75953464
--- Diff: R/pkg/R/backend.R ---
@@ -37,12 +51,42 @@ callJMethod <- function(objId, methodName, ...) {
invokeJava(isStatic = FALSE, objId$id, methodName, ...)
}
-# Call a static method on a specified className
+#' Call Static Java Methods
+#'
+#' Call a static method in the JVM running the Spark driver.
+#'
+#' @param className class containing the static method to invoke.
+#' @param methodName name of static method to invoke.
+#' @param ... parameters to pass to the Java method.
+#' @export
+#' @seealso callJMethod, newJObject
+#' @examples
+#' \dontrun{
+#' sparkR.session() # Need to have a Spark JVM running before calling callJStatic
+#' callJStatic("java.lang.System", "currentTimeMillis")
+#' callJStatic("java.lang.System", "getProperty", "java.home")
+#' }
callJStatic <- function(className, methodName, ...) {
invokeJava(isStatic = TRUE, className, methodName, ...)
}
-# Create a new object of the specified class name
+#' Create Java Objects
+#'
+#' Create a new Java object in the JVM running the Spark driver.
+#'
+#' @param className name of the class to create
+#' @param ... arguments to be passed to the constructor
+#' @export
+#' @seealso callJMethod, callJStatic
--- End diff --
@seealso needs to be `\link{callJMethod}`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14775: [SPARK-16581][SPARKR] Make JVM backend calling functions...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/14775
LGTM. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r76526193
--- Diff: R/pkg/DESCRIPTION ---
@@ -11,7 +11,7 @@ Authors@R: c(person("Shivaram", "Venkataraman", role = c("aut", "cre"),
email = "felixcheung@apache.org"),
person(family = "The Apache Software Foundation", role = c("aut", "cph")))
URL: http://www.apache.org/ http://spark.apache.org/
-BugReports: https://issues.apache.org/jira/secure/CreateIssueDetails!init.jspa?pid=12315420&components=12325400&issuetype=4
+BugReports: http://issues.apache.org/jira/browse/SPARK
--- End diff --
ah, I thought I tested it logged out. You are right, let's scratch that then.
I like the idea with the wiki - though should that mention when to check with user@spark, when to email dev@spark and when to open a JIRA?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r75956062
--- Diff: R/pkg/R/backend.R ---
@@ -37,12 +51,42 @@ callJMethod <- function(objId, methodName, ...) {
invokeJava(isStatic = FALSE, objId$id, methodName, ...)
}
-# Call a static method on a specified className
+#' Call Static Java Methods
+#'
+#' Call a static method in the JVM running the Spark driver.
+#'
+#' @param className class containing the static method to invoke.
+#' @param methodName name of static method to invoke.
+#' @param ... parameters to pass to the Java method.
+#' @export
+#' @seealso callJMethod, newJObject
+#' @examples
+#' \dontrun{
+#' sparkR.session() # Need to have a Spark JVM running before calling callJStatic
+#' callJStatic("java.lang.System", "currentTimeMillis")
+#' callJStatic("java.lang.System", "getProperty", "java.home")
+#' }
callJStatic <- function(className, methodName, ...) {
invokeJava(isStatic = TRUE, className, methodName, ...)
}
-# Create a new object of the specified class name
+#' Create Java Objects
+#'
+#' Create a new Java object in the JVM running the Spark driver.
+#'
+#' @param className name of the class to create
--- End diff --
`Java fully qualified class name`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14775: [SPARK-16581][SPARKR] Make JVM backend calling functions...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14775
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64585/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r76522940
--- Diff: R/pkg/R/backend.R ---
@@ -25,9 +25,23 @@ isInstanceOf <- function(jobj, className) {
callJMethod(cls, "isInstance", jobj)
}
-# Call a Java method named methodName on the object
-# specified by objId. objId should be a "jobj" returned
-# from the SparkRBackend.
+#' Call Java Methods
+#'
+#' Call a Java method in the JVM running the Spark driver.
+#'
+#' @param objId object to invoke the method on. Should be a "jobj" created by newJObject.
--- End diff --
Done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r75955078
--- Diff: R/pkg/R/backend.R ---
@@ -25,9 +25,23 @@ isInstanceOf <- function(jobj, className) {
callJMethod(cls, "isInstance", jobj)
}
-# Call a Java method named methodName on the object
-# specified by objId. objId should be a "jobj" returned
-# from the SparkRBackend.
+#' Call Java Methods
+#'
+#' Call a Java method in the JVM running the Spark driver.
+#'
+#' @param objId object to invoke the method on. Should be a "jobj" created by newJObject.
--- End diff --
if we have a wrapper, it would be better to name this `x`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r76526094
--- Diff: R/pkg/DESCRIPTION ---
@@ -11,7 +11,7 @@ Authors@R: c(person("Shivaram", "Venkataraman", role = c("aut", "cre"),
email = "felixcheung@apache.org"),
person(family = "The Apache Software Foundation", role = c("aut", "cph")))
URL: http://www.apache.org/ http://spark.apache.org/
-BugReports: https://issues.apache.org/jira/secure/CreateIssueDetails!init.jspa?pid=12315420&components=12325400&issuetype=4
+BugReports: http://issues.apache.org/jira/browse/SPARK
--- End diff --
Perhaps that's too long but that opens directly to a form with the component field preset to SparkR - otherwise it seems easy to get lost?
Also BugReports is supposed to be for`bug.report(package = "SparkR")`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14775: [SPARK-16581][SPARKR] Make JVM backend calling functions...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14775
**[Test build #64540 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64540/consoleFull)** for PR 14775 at commit [`448de0c`](https://github.com/apache/spark/commit/448de0c2faf72d3e1fc4d08aa88cd41e8b20c79f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r76526025
--- Diff: R/pkg/R/backend.R ---
@@ -37,12 +51,42 @@ callJMethod <- function(objId, methodName, ...) {
invokeJava(isStatic = FALSE, objId$id, methodName, ...)
}
-# Call a static method on a specified className
+#' Call Static Java Methods
+#'
+#' Call a static method in the JVM running the Spark driver.
+#'
+#' @param className class containing the static method to invoke.
+#' @param methodName name of static method to invoke.
+#' @param ... parameters to pass to the Java method.
+#' @export
+#' @seealso callJMethod, newJObject
+#' @examples
+#' \dontrun{
+#' sparkR.session() # Need to have a Spark JVM running before calling callJStatic
+#' callJStatic("java.lang.System", "currentTimeMillis")
+#' callJStatic("java.lang.System", "getProperty", "java.home")
+#' }
callJStatic <- function(className, methodName, ...) {
invokeJava(isStatic = TRUE, className, methodName, ...)
}
-# Create a new object of the specified class name
+#' Create Java Objects
+#'
+#' Create a new Java object in the JVM running the Spark driver.
+#'
+#' @param className name of the class to create
--- End diff --
I think the behavior could use some explaining and agree that this API is really for advanced or package developers. How about we put a statement or a few words to that effect in the API doc? I think it's ok we don't put a link or describe this in details in the programming guide. Maybe a .md file later?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14775: [SPARK-16581][SPARKR] Make JVM backend calling functions...
Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on the issue:
https://github.com/apache/spark/pull/14775
cc @felixcheung @olarayej
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r76526127
--- Diff: R/pkg/DESCRIPTION ---
@@ -11,7 +11,7 @@ Authors@R: c(person("Shivaram", "Venkataraman", role = c("aut", "cre"),
email = "felixcheung@apache.org"),
person(family = "The Apache Software Foundation", role = c("aut", "cph")))
URL: http://www.apache.org/ http://spark.apache.org/
-BugReports: https://issues.apache.org/jira/secure/CreateIssueDetails!init.jspa?pid=12315420&components=12325400&issuetype=4
+BugReports: http://issues.apache.org/jira/browse/SPARK
--- End diff --
Other than being long that link also doesn't seem to work if you are not logged in (would be good if you can also check this). The other thing we could do is to just link to the wiki page at https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark#ContributingtoSpark-ContributingBugReports -- Do you think that is better ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14775: [SPARK-16581][SPARKR] Make JVM backend calling functions...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14775
**[Test build #64307 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64307/consoleFull)** for PR 14775 at commit [`d267f2f`](https://github.com/apache/spark/commit/d267f2f271aa4e3f7bd8542fa1772ecc978eeee8).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r75956724
--- Diff: R/pkg/R/jobj.R ---
@@ -82,7 +82,20 @@ getClassName.jobj <- function(x) {
callJMethod(cls, "getName")
}
-cleanup.jobj <- function(jobj) {
+#' Garbage collect Java Objects
+#'
+#' Garbage collect an object allocated on Spark driver JVM heap.
+#'
+#' cleanup.jobj is a low level method that lets developers manually garbage collect objects
+#' allocated using newJObject. This is only to be used for advanced use cases as objects allocated
+#' on the JVM heap are automatically garbage collected when the corresponding R reference goes out
+#' of scope.
+#'
+#' @param x the Java object that should be garbage collected.
+#' @note cleanup.jobj since 2.0.1
--- End diff --
@aliases
@return?
@rdname
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r75953989
--- Diff: R/pkg/R/jobj.R ---
@@ -82,7 +82,20 @@ getClassName.jobj <- function(x) {
callJMethod(cls, "getName")
}
-cleanup.jobj <- function(jobj) {
+#' Garbage collect Java Objects
+#'
+#' Garbage collect an object allocated on Spark driver JVM heap.
+#'
+#' cleanup.jobj is a low level method that lets developers manually garbage collect objects
+#' allocated using newJObject. This is only to be used for advanced use cases as objects allocated
+#' on the JVM heap are automatically garbage collected when the corresponding R reference goes out
+#' of scope.
+#'
+#' @param x the Java object that should be garbage collected.
+#' @note cleanup.jobj since 2.0.1
+#' @export
+cleanup.jobj <- function(x) {
+ jobj <- x
if (isValidJobj(jobj)) {
--- End diff --
isValidJobj doesn't currently check, I think we should add a `if (class(jobj) == 'jobj')` before that
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r75953769
--- Diff: R/pkg/R/jobj.R ---
@@ -82,7 +82,20 @@ getClassName.jobj <- function(x) {
callJMethod(cls, "getName")
}
-cleanup.jobj <- function(jobj) {
+#' Garbage collect Java Objects
+#'
+#' Garbage collect an object allocated on Spark driver JVM heap.
+#'
+#' cleanup.jobj is a low level method that lets developers manually garbage collect objects
+#' allocated using newJObject. This is only to be used for advanced use cases as objects allocated
+#' on the JVM heap are automatically garbage collected when the corresponding R reference goes out
+#' of scope.
+#'
+#' @param x the Java object that should be garbage collected.
+#' @note cleanup.jobj since 2.0.1
+#' @export
+cleanup.jobj <- function(x) {
--- End diff --
Is `cleanup` a good descriptive name for R audience? Would `gc.jobj()` better?
http://stackoverflow.com/questions/8813753/what-is-the-difference-between-gc-and-rm
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14775: [SPARK-16581][SPARKR] Make JVM backend calling functions...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14775
**[Test build #64540 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64540/consoleFull)** for PR 14775 at commit [`448de0c`](https://github.com/apache/spark/commit/448de0c2faf72d3e1fc4d08aa88cd41e8b20c79f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14775: [SPARK-16581][SPARKR] Make JVM backend calling functions...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14775
**[Test build #64539 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64539/consoleFull)** for PR 14775 at commit [`0959208`](https://github.com/apache/spark/commit/0959208424e085cc3769e03d4b67779e08296009).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14775: [SPARK-16581][SPARKR] Make JVM backend calling functions...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14775
**[Test build #64307 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64307/consoleFull)** for PR 14775 at commit [`d267f2f`](https://github.com/apache/spark/commit/d267f2f271aa4e3f7bd8542fa1772ecc978eeee8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14775: [SPARK-16581][SPARKR] Make JVM backend calling functions...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14775
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r76654243
--- Diff: R/pkg/R/backend.R ---
@@ -37,12 +51,42 @@ callJMethod <- function(objId, methodName, ...) {
invokeJava(isStatic = FALSE, objId$id, methodName, ...)
}
-# Call a static method on a specified className
+#' Call Static Java Methods
+#'
+#' Call a static method in the JVM running the Spark driver.
+#'
+#' @param className class containing the static method to invoke.
+#' @param methodName name of static method to invoke.
+#' @param ... parameters to pass to the Java method.
+#' @export
+#' @seealso callJMethod, newJObject
+#' @examples
+#' \dontrun{
+#' sparkR.session() # Need to have a Spark JVM running before calling callJStatic
+#' callJStatic("java.lang.System", "currentTimeMillis")
+#' callJStatic("java.lang.System", "getProperty", "java.home")
+#' }
callJStatic <- function(className, methodName, ...) {
invokeJava(isStatic = TRUE, className, methodName, ...)
}
-# Create a new object of the specified class name
+#' Create Java Objects
+#'
+#' Create a new Java object in the JVM running the Spark driver.
+#'
+#' @param className name of the class to create
--- End diff --
I added some comments in a `Details` section. Let me know if it reads ok. I agree that having a md file might be useful in the long run.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r76444282
--- Diff: R/pkg/NAMESPACE ---
@@ -363,4 +363,9 @@ S3method(structField, jobj)
S3method(structType, jobj)
S3method(structType, structField)
+export("newJObject")
+export("callJMethod")
+export("callJStatic")
+export("cleanup.jobj")
--- End diff --
Thanks @sun-rui and @aloknsingh - I'll update the PR today
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14775: [SPARK-16581][SPARKR] Make JVM backend calling functions...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14775
**[Test build #64585 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64585/consoleFull)** for PR 14775 at commit [`d1ec80b`](https://github.com/apache/spark/commit/d1ec80b3a57b0610ce0367447e9802923ad02ae9).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r76522977
--- Diff: R/pkg/R/backend.R ---
@@ -37,12 +51,42 @@ callJMethod <- function(objId, methodName, ...) {
invokeJava(isStatic = FALSE, objId$id, methodName, ...)
}
-# Call a static method on a specified className
+#' Call Static Java Methods
+#'
+#' Call a static method in the JVM running the Spark driver.
+#'
+#' @param className class containing the static method to invoke.
+#' @param methodName name of static method to invoke.
+#' @param ... parameters to pass to the Java method.
+#' @export
+#' @seealso callJMethod, newJObject
+#' @examples
+#' \dontrun{
+#' sparkR.session() # Need to have a Spark JVM running before calling callJStatic
+#' callJStatic("java.lang.System", "currentTimeMillis")
+#' callJStatic("java.lang.System", "getProperty", "java.home")
+#' }
callJStatic <- function(className, methodName, ...) {
invokeJava(isStatic = TRUE, className, methodName, ...)
}
-# Create a new object of the specified class name
+#' Create Java Objects
+#'
+#' Create a new Java object in the JVM running the Spark driver.
+#'
+#' @param className name of the class to create
--- End diff --
I tried to qualify this in the comment on what gets returned. The trouble is that we dont have an externally visible documentation of what types will get converted vs. what will not. Also I think this is bound to change with versions (like the SQL decimal change for example).
However this is a very low level API that is only for advanced developers. So I wonder if we should just leave a pointer to the source file ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r75955890
--- Diff: R/pkg/R/backend.R ---
@@ -37,12 +51,42 @@ callJMethod <- function(objId, methodName, ...) {
invokeJava(isStatic = FALSE, objId$id, methodName, ...)
}
-# Call a static method on a specified className
+#' Call Static Java Methods
+#'
+#' Call a static method in the JVM running the Spark driver.
+#'
+#' @param className class containing the static method to invoke.
--- End diff --
Java fully qualified class name
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r75956948
--- Diff: R/pkg/R/backend.R ---
@@ -25,9 +25,23 @@ isInstanceOf <- function(jobj, className) {
callJMethod(cls, "isInstance", jobj)
}
-# Call a Java method named methodName on the object
-# specified by objId. objId should be a "jobj" returned
-# from the SparkRBackend.
+#' Call Java Methods
+#'
+#' Call a Java method in the JVM running the Spark driver.
+#'
+#' @param objId object to invoke the method on. Should be a "jobj" created by newJObject.
+#' @param methodName method name to call.
+#' @param ... parameters to pass to the Java method.
+#' @export
+#' @seealso callJStatic, newJObject
--- End diff --
ditto
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14775: [SPARK-16581][SPARKR] Make JVM backend calling functions...
Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on the issue:
https://github.com/apache/spark/pull/14775
Thanks @felixcheung - Addressed both the comments. Let me know if this looks good.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r76522987
--- Diff: R/pkg/R/jobj.R ---
@@ -82,7 +82,20 @@ getClassName.jobj <- function(x) {
callJMethod(cls, "getName")
}
-cleanup.jobj <- function(jobj) {
+#' Garbage collect Java Objects
+#'
+#' Garbage collect an object allocated on Spark driver JVM heap.
+#'
+#' cleanup.jobj is a low level method that lets developers manually garbage collect objects
+#' allocated using newJObject. This is only to be used for advanced use cases as objects allocated
+#' on the JVM heap are automatically garbage collected when the corresponding R reference goes out
+#' of scope.
+#'
+#' @param x the Java object that should be garbage collected.
+#' @note cleanup.jobj since 2.0.1
--- End diff --
This is removed now
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r76522952
--- Diff: R/pkg/R/backend.R ---
@@ -25,9 +25,23 @@ isInstanceOf <- function(jobj, className) {
callJMethod(cls, "isInstance", jobj)
}
-# Call a Java method named methodName on the object
-# specified by objId. objId should be a "jobj" returned
-# from the SparkRBackend.
+#' Call Java Methods
+#'
+#' Call a Java method in the JVM running the Spark driver.
+#'
+#' @param objId object to invoke the method on. Should be a "jobj" created by newJObject.
+#' @param methodName method name to call.
+#' @param ... parameters to pass to the Java method.
+#' @export
+#' @seealso callJStatic, newJObject
+#' @examples
--- End diff --
add the `@note` and `@return` to all the methods. I am not sure we need the `@aliases` or `@rdname` because these are S3 methods ? The rdnames come out as `sparkR.callJMethod.rd` etc. and look fine to me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r76522957
--- Diff: R/pkg/R/backend.R ---
@@ -37,12 +51,42 @@ callJMethod <- function(objId, methodName, ...) {
invokeJava(isStatic = FALSE, objId$id, methodName, ...)
}
-# Call a static method on a specified className
+#' Call Static Java Methods
+#'
+#' Call a static method in the JVM running the Spark driver.
+#'
+#' @param className class containing the static method to invoke.
+#' @param methodName name of static method to invoke.
+#' @param ... parameters to pass to the Java method.
+#' @export
+#' @seealso callJMethod, newJObject
--- End diff --
Done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r75955453
--- Diff: R/pkg/R/backend.R ---
@@ -25,9 +25,23 @@ isInstanceOf <- function(jobj, className) {
callJMethod(cls, "isInstance", jobj)
}
-# Call a Java method named methodName on the object
-# specified by objId. objId should be a "jobj" returned
-# from the SparkRBackend.
+#' Call Java Methods
+#'
+#' Call a Java method in the JVM running the Spark driver.
+#'
+#' @param objId object to invoke the method on. Should be a "jobj" created by newJObject.
+#' @param methodName method name to call.
+#' @param ... parameters to pass to the Java method.
+#' @export
+#' @seealso callJStatic, newJObject
+#' @examples
--- End diff --
need @aliases, @note since 2.0.1?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r76522986
--- Diff: R/pkg/R/backend.R ---
@@ -37,12 +51,42 @@ callJMethod <- function(objId, methodName, ...) {
invokeJava(isStatic = FALSE, objId$id, methodName, ...)
}
-# Call a static method on a specified className
+#' Call Static Java Methods
+#'
+#' Call a static method in the JVM running the Spark driver.
+#'
+#' @param className class containing the static method to invoke.
+#' @param methodName name of static method to invoke.
+#' @param ... parameters to pass to the Java method.
+#' @export
+#' @seealso callJMethod, newJObject
+#' @examples
+#' \dontrun{
+#' sparkR.session() # Need to have a Spark JVM running before calling callJStatic
+#' callJStatic("java.lang.System", "currentTimeMillis")
+#' callJStatic("java.lang.System", "getProperty", "java.home")
+#' }
callJStatic <- function(className, methodName, ...) {
invokeJava(isStatic = TRUE, className, methodName, ...)
}
-# Create a new object of the specified class name
+#' Create Java Objects
+#'
+#' Create a new Java object in the JVM running the Spark driver.
+#'
+#' @param className name of the class to create
+#' @param ... arguments to be passed to the constructor
+#' @export
+#' @seealso callJMethod, callJStatic
--- End diff --
Done the `seealso` and `return` - Lets discuss the `alias`, `rdname` in the other thread
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14775: [SPARK-16581][SPARKR] Make JVM backend calling functions...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/14775
github seems to like to hide stuff.
FYI, this comment https://github.com/apache/spark/pull/14775#discussion_r76526025
and this https://github.com/apache/spark/pull/14775#discussion_r76526168
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r75955803
--- Diff: R/pkg/R/backend.R ---
@@ -25,9 +25,23 @@ isInstanceOf <- function(jobj, className) {
callJMethod(cls, "isInstance", jobj)
}
-# Call a Java method named methodName on the object
-# specified by objId. objId should be a "jobj" returned
-# from the SparkRBackend.
+#' Call Java Methods
+#'
+#' Call a Java method in the JVM running the Spark driver.
+#'
+#' @param objId object to invoke the method on. Should be a "jobj" created by newJObject.
+#' @param methodName method name to call.
+#' @param ... parameters to pass to the Java method.
+#' @export
+#' @seealso callJStatic, newJObject
+#' @examples
--- End diff --
should this return `invisible()` for exported version?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r75956166
--- Diff: R/pkg/R/backend.R ---
@@ -37,12 +51,42 @@ callJMethod <- function(objId, methodName, ...) {
invokeJava(isStatic = FALSE, objId$id, methodName, ...)
}
-# Call a static method on a specified className
+#' Call Static Java Methods
+#'
+#' Call a static method in the JVM running the Spark driver.
+#'
+#' @param className class containing the static method to invoke.
+#' @param methodName name of static method to invoke.
+#' @param ... parameters to pass to the Java method.
+#' @export
+#' @seealso callJMethod, newJObject
+#' @examples
+#' \dontrun{
+#' sparkR.session() # Need to have a Spark JVM running before calling callJStatic
+#' callJStatic("java.lang.System", "currentTimeMillis")
+#' callJStatic("java.lang.System", "getProperty", "java.home")
+#' }
callJStatic <- function(className, methodName, ...) {
invokeJava(isStatic = TRUE, className, methodName, ...)
}
-# Create a new object of the specified class name
+#' Create Java Objects
+#'
+#' Create a new Java object in the JVM running the Spark driver.
+#'
+#' @param className name of the class to create
+#' @param ... arguments to be passed to the constructor
+#' @export
+#' @seealso callJMethod, callJStatic
--- End diff --
@aliases
@return
@rdname
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r76654300
--- Diff: R/pkg/R/backend.R ---
@@ -25,9 +25,23 @@ isInstanceOf <- function(jobj, className) {
callJMethod(cls, "isInstance", jobj)
}
-# Call a Java method named methodName on the object
-# specified by objId. objId should be a "jobj" returned
-# from the SparkRBackend.
+#' Call Java Methods
+#'
+#' Call a Java method in the JVM running the Spark driver.
+#'
+#' @param objId object to invoke the method on. Should be a "jobj" created by newJObject.
+#' @param methodName method name to call.
+#' @param ... parameters to pass to the Java method.
+#' @export
+#' @seealso callJStatic, newJObject
+#' @examples
--- End diff --
Added `rdname` for all 3 methods now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14775: [SPARK-16581][SPARKR] Make JVM backend calling functions...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14775
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r76526121
--- Diff: R/pkg/R/jobj.R ---
@@ -82,7 +82,20 @@ getClassName.jobj <- function(x) {
callJMethod(cls, "getName")
}
-cleanup.jobj <- function(jobj) {
+#' Garbage collect Java Objects
+#'
+#' Garbage collect an object allocated on Spark driver JVM heap.
+#'
+#' cleanup.jobj is a low level method that lets developers manually garbage collect objects
+#' allocated using newJObject. This is only to be used for advanced use cases as objects allocated
+#' on the JVM heap are automatically garbage collected when the corresponding R reference goes out
+#' of scope.
+#'
+#' @param x the Java object that should be garbage collected.
+#' @note cleanup.jobj since 2.0.1
+#' @export
+cleanup.jobj <- function(x) {
+ jobj <- x
if (isValidJobj(jobj)) {
--- End diff --
I think we are good for now if we are not exporting `cleanup.jobj`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r75953330
--- Diff: R/pkg/R/backend.R ---
@@ -37,12 +51,42 @@ callJMethod <- function(objId, methodName, ...) {
invokeJava(isStatic = FALSE, objId$id, methodName, ...)
}
-# Call a static method on a specified className
+#' Call Static Java Methods
+#'
+#' Call a static method in the JVM running the Spark driver.
+#'
+#' @param className class containing the static method to invoke.
+#' @param methodName name of static method to invoke.
+#' @param ... parameters to pass to the Java method.
+#' @export
+#' @seealso callJMethod, newJObject
+#' @examples
+#' \dontrun{
+#' sparkR.session() # Need to have a Spark JVM running before calling callJStatic
+#' callJStatic("java.lang.System", "currentTimeMillis")
+#' callJStatic("java.lang.System", "getProperty", "java.home")
+#' }
callJStatic <- function(className, methodName, ...) {
invokeJava(isStatic = TRUE, className, methodName, ...)
}
-# Create a new object of the specified class name
+#' Create Java Objects
+#'
+#' Create a new Java object in the JVM running the Spark driver.
+#'
+#' @param className name of the class to create
--- End diff --
when certain JVM types are created because our [SerDe](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/api/r/SerDe.scala#L367) they don't always come back as jobj - should we call this out?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14775: [SPARK-16581][SPARKR] Make JVM backend calling functions...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14775
**[Test build #64585 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64585/consoleFull)** for PR 14775 at commit [`d1ec80b`](https://github.com/apache/spark/commit/d1ec80b3a57b0610ce0367447e9802923ad02ae9).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r76522943
--- Diff: R/pkg/R/backend.R ---
@@ -25,9 +25,23 @@ isInstanceOf <- function(jobj, className) {
callJMethod(cls, "isInstance", jobj)
}
-# Call a Java method named methodName on the object
-# specified by objId. objId should be a "jobj" returned
-# from the SparkRBackend.
+#' Call Java Methods
+#'
+#' Call a Java method in the JVM running the Spark driver.
+#'
+#' @param objId object to invoke the method on. Should be a "jobj" created by newJObject.
+#' @param methodName method name to call.
+#' @param ... parameters to pass to the Java method.
+#' @export
+#' @seealso callJStatic, newJObject
--- End diff --
Done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r75956533
--- Diff: R/pkg/inst/tests/testthat/test_jvm_api.R ---
@@ -0,0 +1,41 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+context("JVM API")
+
+sparkSession <- sparkR.session(enableHiveSupport = FALSE)
+
+test_that("Create and call methods on object", {
+ jarr <- newJObject("java.util.ArrayList")
+ # Add an element to the array
+ callJMethod(jarr, "add", 1L)
+ # Check if get returns the same element
+ expect_equal(callJMethod(jarr, "get", 0L), 1L)
+})
+
+test_that("Call static methods", {
+ # Convert a boolean to a string
+ strTrue <- callJStatic("java.lang.String", "valueOf", TRUE)
+ expect_equal(strTrue, "true")
+})
+
+test_that("Manually garbage collect objects", {
+ jarr <- newJObject("java.util.ArrayList")
+ cleanup.jobj(jarr)
+ # Using a jobj after GC should throw an error
+ expect_error(print(jarr), "Error in invokeJava.*")
+})
--- End diff --
add `sparkR.session.stop()`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14775: [SPARK-16581][SPARKR] Make JVM backend calling functions...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14775
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14775: [SPARK-16581][SPARKR] Make JVM backend calling functions...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14775
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64539/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r76645788
--- Diff: R/pkg/DESCRIPTION ---
@@ -11,7 +11,7 @@ Authors@R: c(person("Shivaram", "Venkataraman", role = c("aut", "cre"),
email = "felixcheung@apache.org"),
person(family = "The Apache Software Foundation", role = c("aut", "cph")))
URL: http://www.apache.org/ http://spark.apache.org/
-BugReports: https://issues.apache.org/jira/secure/CreateIssueDetails!init.jspa?pid=12315420&components=12325400&issuetype=4
+BugReports: http://issues.apache.org/jira/browse/SPARK
--- End diff --
I updated the wiki - Let me know if it looks better or if you have other suggestions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r76526168
--- Diff: R/pkg/R/backend.R ---
@@ -25,9 +25,23 @@ isInstanceOf <- function(jobj, className) {
callJMethod(cls, "isInstance", jobj)
}
-# Call a Java method named methodName on the object
-# specified by objId. objId should be a "jobj" returned
-# from the SparkRBackend.
+#' Call Java Methods
+#'
+#' Call a Java method in the JVM running the Spark driver.
+#'
+#' @param objId object to invoke the method on. Should be a "jobj" created by newJObject.
+#' @param methodName method name to call.
+#' @param ... parameters to pass to the Java method.
+#' @export
+#' @seealso callJStatic, newJObject
+#' @examples
--- End diff --
roxygen2 infers the rdname pretty well, so I agree that is largely optional.
From a best practice/guideline point-of-view though I think it'd be good to have `@rdname` because it would make it clear which Rd and doc page it goes to. I think it would get people to think about it.
For example, instead of `n`->`n` by default we want `n`->`count` since `n` is an alias of `count`.
As for @aliases, agree we don't need it. On a related note, I think I'm seeing some issues with it in other places we have that I want to look into shortly. Let me check back with you later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by aloknsingh <gi...@git.apache.org>.
Github user aloknsingh commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r76381368
--- Diff: R/pkg/NAMESPACE ---
@@ -363,4 +363,9 @@ S3method(structField, jobj)
S3method(structType, jobj)
S3method(structType, structField)
+export("newJObject")
+export("callJMethod")
+export("callJStatic")
+export("cleanup.jobj")
--- End diff --
Hi @sun-rui
Thanks for pointing this out. I look at the code and you are right in mentioning that the jobj has the cleanup.jobj registered.
Initially, we were under the assumption that we have to register to the the object created otherwise the jvm (running scala) won't cleanup our obj, as there is the ref in the SparkR.
In summary,
1) For the systemML, with the information you had mentioned, we would be fine with if it is made private also.
2) Also as you had mentioned, if package writer (on the top of sparkR) feels that they have absolutely run the cleanup, they can call the R's explicit "gc".
thanks,
Alok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14775: [SPARK-16581][SPARKR] Make JVM backend calling functions...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14775
**[Test build #64539 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64539/consoleFull)** for PR 14775 at commit [`0959208`](https://github.com/apache/spark/commit/0959208424e085cc3769e03d4b67779e08296009).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r76522998
--- Diff: R/pkg/inst/tests/testthat/test_jvm_api.R ---
@@ -0,0 +1,41 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+context("JVM API")
+
+sparkSession <- sparkR.session(enableHiveSupport = FALSE)
+
+test_that("Create and call methods on object", {
+ jarr <- newJObject("java.util.ArrayList")
+ # Add an element to the array
+ callJMethod(jarr, "add", 1L)
+ # Check if get returns the same element
+ expect_equal(callJMethod(jarr, "get", 0L), 1L)
+})
+
+test_that("Call static methods", {
+ # Convert a boolean to a string
+ strTrue <- callJStatic("java.lang.String", "valueOf", TRUE)
+ expect_equal(strTrue, "true")
+})
+
+test_that("Manually garbage collect objects", {
+ jarr <- newJObject("java.util.ArrayList")
+ cleanup.jobj(jarr)
+ # Using a jobj after GC should throw an error
+ expect_error(print(jarr), "Error in invokeJava.*")
+})
--- End diff --
Done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14775: [SPARK-16581][SPARKR] Make JVM backend calling functions...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14775
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64307/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by sun-rui <gi...@git.apache.org>.
Github user sun-rui commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r76008164
--- Diff: R/pkg/NAMESPACE ---
@@ -363,4 +363,9 @@ S3method(structField, jobj)
S3method(structType, jobj)
S3method(structType, structField)
+export("newJObject")
+export("callJMethod")
+export("callJStatic")
+export("cleanup.jobj")
--- End diff --
@aloknsingh
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by sun-rui <gi...@git.apache.org>.
Github user sun-rui commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r76007335
--- Diff: R/pkg/NAMESPACE ---
@@ -363,4 +363,9 @@ S3method(structField, jobj)
S3method(structType, jobj)
S3method(structType, structField)
+export("newJObject")
+export("callJMethod")
+export("callJStatic")
+export("cleanup.jobj")
--- End diff --
I am a little confused about the discussion at https://issues.apache.org/jira/browse/SPARK-16611:
````
d) cleanup.jobj:
SystemML uses the MLContext and matrixCharacteristic class that is instantiated in JVM and whose object ref is kept alive in the sparkR and later when systemML has done it\u2019s computation. we cleanup the objects. The way we achieve it using the References classes in R and use it\u2019s finalize method to register the cleanup.jobj once we have created the jobj via newJObject(\u201csysml.class\u201d)
````
It seems that systemML still uses automatic GC to cleanup jobjs as it is said that "use it\u2019s finalize method to register the cleanup.jobj ". But a jobj already has cleanup.jobj registered as its finalizer.
If my understanding is correct, I don't see strong reason for exposing cleanup.jobj. If active cleanup is demanded, a call to "gc" can be made?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/14775
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14775: [SPARK-16581][SPARKR] Make JVM backend calling functions...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/14775
I think the downside of naming them as-is and keeping the signature (`...` at the end) are that it would be very hard to change or add to the signature later on (say, to add a context/context_id)
How about naming them `sparkR.callJMethod` or similar for the exported versions to wrap around the internal ones?
this way we don't have to break internal stuff and could just add a new method if/when we want.
we could also additional parameter checks or translation as needed that might not apply to internal stuff, which could also mitigate risk
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14775: [SPARK-16581][SPARKR] Make JVM backend calling functions...
Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on the issue:
https://github.com/apache/spark/pull/14775
@felixcheung Good point about having a wrapper -- That will make it easier to update the methods going forward. I added a new file `jvm.R` with the wrapper functions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r75956901
--- Diff: R/pkg/R/backend.R ---
@@ -37,12 +51,42 @@ callJMethod <- function(objId, methodName, ...) {
invokeJava(isStatic = FALSE, objId$id, methodName, ...)
}
-# Call a static method on a specified className
+#' Call Static Java Methods
+#'
+#' Call a static method in the JVM running the Spark driver.
+#'
+#' @param className class containing the static method to invoke.
+#' @param methodName name of static method to invoke.
+#' @param ... parameters to pass to the Java method.
+#' @export
+#' @seealso callJMethod, newJObject
--- End diff --
ditto
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14775: [SPARK-16581][SPARKR] Make JVM backend calling fu...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/14775#discussion_r75955348
--- Diff: R/pkg/R/backend.R ---
@@ -25,9 +25,23 @@ isInstanceOf <- function(jobj, className) {
callJMethod(cls, "isInstance", jobj)
}
-# Call a Java method named methodName on the object
-# specified by objId. objId should be a "jobj" returned
-# from the SparkRBackend.
+#' Call Java Methods
+#'
+#' Call a Java method in the JVM running the Spark driver.
+#'
+#' @param objId object to invoke the method on. Should be a "jobj" created by newJObject.
+#' @param methodName method name to call.
--- End diff --
`@param methodName character method name to call. this should be a Java fully qualified class name.`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14775: [SPARK-16581][SPARKR] Make JVM backend calling functions...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14775
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14775: [SPARK-16581][SPARKR] Make JVM backend calling functions...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14775
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64540/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org