You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by shivaram <gi...@git.apache.org> on 2017/10/31 20:20:52 UTC
[GitHub] spark pull request #19624: [SPARKR][SPARK-22315] Warn is SparkR package vers...
GitHub user shivaram opened a pull request:
https://github.com/apache/spark/pull/19624
[SPARKR][SPARK-22315] Warn is SparkR package version doesn't match SparkContext
## What changes were proposed in this pull request?
This PR adds a check between the R package version used and the version reported by SparkContext running in the JVM. The goal here is to warn users when they have a R package downloaded from CRAN and are using that to connect to an existing Spark cluster.
This is raised as a warning rather than an error as users might want to use patch versions interchangeably (e.g., 2.1.3 with 2.1.2 etc.)
## How was this patch tested?
Manually by changing the `DESCRIPTION` file
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/shivaram/spark-1 sparkr-version-check
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19624.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19624
----
commit 6f6696762a8e779527339360c28f134da36940d6
Author: Shivaram Venkataraman <sh...@cs.berkeley.edu>
Date: 2017-10-30T02:30:25Z
Warn if R pkg version doesn't match JVM version
commit 680a38ccd106145cd7dd72c10ed9f2b1b41b18f6
Author: Shivaram Venkataraman <sh...@cs.berkeley.edu>
Date: 2017-10-31T20:11:46Z
Remove snapshot before comparing versions
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19624: [SPARKR][SPARK-22315] Warn if SparkR package vers...
Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/19624#discussion_r148320092
--- Diff: R/pkg/R/sparkR.R ---
@@ -420,6 +420,18 @@ sparkR.session <- function(
enableHiveSupport)
assign(".sparkRsession", sparkSession, envir = .sparkREnv)
}
+
+ # Check if version number of SparkSession matches version number of SparkR package
+ jvmVersion <- callJMethod(sparkSession, "version")
+ # Remove -SNAPSHOT from jvm versions
+ jvmVersion <- gsub("-SNAPSHOT", "", jvmVersion)
+ rPackageVersion <- paste0(packageVersion("SparkR"))
+
+ if (jvmVersion != rPackageVersion) {
+ warning(paste("Version mismatch between Spark JVM and SparkR package. JVM version was",
+ jvmVersion, ", while R package version was", rPackageVersion))
--- End diff --
I wonder if that would be more confusing as somebody might think they just need to remove the `-snapshot` ?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19624: [SPARKR][SPARK-22315] Warn if SparkR package version doe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19624
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83276/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19624: [SPARKR][SPARK-22315] Warn if SparkR package version doe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19624
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19624: [SPARKR][SPARK-22315] Warn if SparkR package version doe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19624
**[Test build #83467 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83467/testReport)** for PR 19624 at commit [`cc6b540`](https://github.com/apache/spark/commit/cc6b540842bf41ade19650c7b5fe289f8fc783b7).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19624: [SPARKR][SPARK-22315] Warn if SparkR package version doe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19624
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19624: [SPARKR][SPARK-22315] Warn if SparkR package vers...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/19624#discussion_r148192869
--- Diff: R/pkg/R/sparkR.R ---
@@ -420,6 +420,18 @@ sparkR.session <- function(
enableHiveSupport)
assign(".sparkRsession", sparkSession, envir = .sparkREnv)
}
+
+ # Check if version number of SparkSession matches version number of SparkR package
+ jvmVersion <- callJMethod(sparkSession, "version")
+ # Remove -SNAPSHOT from jvm versions
+ jvmVersion <- gsub("-SNAPSHOT", "", jvmVersion)
+ rPackageVersion <- paste0(packageVersion("SparkR"))
+
+ if (jvmVersion != rPackageVersion) {
+ warning(paste("Version mismatch between Spark JVM and SparkR package. JVM version was",
+ jvmVersion, ", while R package version was", rPackageVersion))
--- End diff --
not sure if it's important, perhaps print the original version (ie. with -SNAPSHOT)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19624: [SPARKR][SPARK-22315] Warn if SparkR package version doe...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/19624
Maybe but it will say
Version mismatch between Spark JVM and SparkR package. JVM version was 2.3.0-SNAPSHOT, while R package version was 2.1.2
I think it will be clear the numbers are different.
If they are both 2.3.0 it won't show the warning.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19624: [SPARKR][SPARK-22315] Warn if SparkR package version doe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19624
**[Test build #83276 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83276/testReport)** for PR 19624 at commit [`680a38c`](https://github.com/apache/spark/commit/680a38ccd106145cd7dd72c10ed9f2b1b41b18f6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19624: [SPARKR][SPARK-22315] Warn if SparkR package version doe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19624
**[Test build #83467 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83467/testReport)** for PR 19624 at commit [`cc6b540`](https://github.com/apache/spark/commit/cc6b540842bf41ade19650c7b5fe289f8fc783b7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19624: [SPARKR][SPARK-22315] Warn if SparkR package version doe...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19624
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83467/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19624: [SPARKR][SPARK-22315] Warn if SparkR package version doe...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/19624
Maybe but it will say
`Version mismatch between Spark JVM and SparkR package. JVM version was 2.3.0-SNAPSHOT, while R package version was 2.1.2`
I think it will be clear the numbers are different.
If they are both 2.3.0 it won't show the warning.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19624: [SPARKR][SPARK-22315] Warn if SparkR package version doe...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19624
**[Test build #83276 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83276/testReport)** for PR 19624 at commit [`680a38c`](https://github.com/apache/spark/commit/680a38ccd106145cd7dd72c10ed9f2b1b41b18f6).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19624: [SPARKR][SPARK-22315] Warn if SparkR package vers...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/19624
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org