You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sh...@apache.org on 2017/11/06 16:58:48 UTC
spark git commit: [SPARK-22315][SPARKR] Warn if SparkR package
version doesn't match SparkContext
Repository: spark
Updated Branches:
refs/heads/master c7f38e5ad -> 65a8bf603
[SPARK-22315][SPARKR] Warn if SparkR package version doesn't match SparkContext
## What changes were proposed in this pull request?
This PR adds a check between the R package version used and the version reported by SparkContext running in the JVM. The goal here is to warn users when they have a R package downloaded from CRAN and are using that to connect to an existing Spark cluster.
This is raised as a warning rather than an error as users might want to use patch versions interchangeably (e.g., 2.1.3 with 2.1.2 etc.)
## How was this patch tested?
Manually by changing the `DESCRIPTION` file
Author: Shivaram Venkataraman <sh...@cs.berkeley.edu>
Closes #19624 from shivaram/sparkr-version-check.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/65a8bf60
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/65a8bf60
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/65a8bf60
Branch: refs/heads/master
Commit: 65a8bf6036fe41a53b4b1e4298fa35d7fa4e9970
Parents: c7f38e5
Author: Shivaram Venkataraman <sh...@cs.berkeley.edu>
Authored: Mon Nov 6 08:58:42 2017 -0800
Committer: Shivaram Venkataraman <sh...@cs.berkeley.edu>
Committed: Mon Nov 6 08:58:42 2017 -0800
----------------------------------------------------------------------
R/pkg/R/sparkR.R | 12 ++++++++++++
1 file changed, 12 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/65a8bf60/R/pkg/R/sparkR.R
----------------------------------------------------------------------
diff --git a/R/pkg/R/sparkR.R b/R/pkg/R/sparkR.R
index 81507ea..fb5f1d2 100644
--- a/R/pkg/R/sparkR.R
+++ b/R/pkg/R/sparkR.R
@@ -420,6 +420,18 @@ sparkR.session <- function(
enableHiveSupport)
assign(".sparkRsession", sparkSession, envir = .sparkREnv)
}
+
+ # Check if version number of SparkSession matches version number of SparkR package
+ jvmVersion <- callJMethod(sparkSession, "version")
+ # Remove -SNAPSHOT from jvm versions
+ jvmVersionStrip <- gsub("-SNAPSHOT", "", jvmVersion)
+ rPackageVersion <- paste0(packageVersion("SparkR"))
+
+ if (jvmVersionStrip != rPackageVersion) {
+ warning(paste("Version mismatch between Spark JVM and SparkR package. JVM version was",
+ jvmVersion, ", while R package version was", rPackageVersion))
+ }
+
sparkSession
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org