You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sh...@apache.org on 2015/12/16 19:32:35 UTC
spark git commit: [SPARK-12318][SPARKR] Save mode in SparkR should be
error by default
Repository: spark
Updated Branches:
refs/heads/master 54c512ba9 -> 2eb5af5f0
[SPARK-12318][SPARKR] Save mode in SparkR should be error by default
shivaram Please help review.
Author: Jeff Zhang <zj...@apache.org>
Closes #10290 from zjffdu/SPARK-12318.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/2eb5af5f
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/2eb5af5f
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/2eb5af5f
Branch: refs/heads/master
Commit: 2eb5af5f0d3c424dc617bb1a18dd0210ea9ba0bc
Parents: 54c512b
Author: Jeff Zhang <zj...@apache.org>
Authored: Wed Dec 16 10:32:32 2015 -0800
Committer: Shivaram Venkataraman <sh...@cs.berkeley.edu>
Committed: Wed Dec 16 10:32:32 2015 -0800
----------------------------------------------------------------------
R/pkg/R/DataFrame.R | 10 +++++-----
docs/sparkr.md | 9 ++++++++-
2 files changed, 13 insertions(+), 6 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/2eb5af5f/R/pkg/R/DataFrame.R
----------------------------------------------------------------------
diff --git a/R/pkg/R/DataFrame.R b/R/pkg/R/DataFrame.R
index 764597d..380a13f 100644
--- a/R/pkg/R/DataFrame.R
+++ b/R/pkg/R/DataFrame.R
@@ -1886,7 +1886,7 @@ setMethod("except",
#' @param df A SparkSQL DataFrame
#' @param path A name for the table
#' @param source A name for external data source
-#' @param mode One of 'append', 'overwrite', 'error', 'ignore' save mode
+#' @param mode One of 'append', 'overwrite', 'error', 'ignore' save mode (it is 'error' by default)
#'
#' @family DataFrame functions
#' @rdname write.df
@@ -1903,7 +1903,7 @@ setMethod("except",
#' }
setMethod("write.df",
signature(df = "DataFrame", path = "character"),
- function(df, path, source = NULL, mode = "append", ...){
+ function(df, path, source = NULL, mode = "error", ...){
if (is.null(source)) {
sqlContext <- get(".sparkRSQLsc", envir = .sparkREnv)
source <- callJMethod(sqlContext, "getConf", "spark.sql.sources.default",
@@ -1928,7 +1928,7 @@ setMethod("write.df",
#' @export
setMethod("saveDF",
signature(df = "DataFrame", path = "character"),
- function(df, path, source = NULL, mode = "append", ...){
+ function(df, path, source = NULL, mode = "error", ...){
write.df(df, path, source, mode, ...)
})
@@ -1951,7 +1951,7 @@ setMethod("saveDF",
#' @param df A SparkSQL DataFrame
#' @param tableName A name for the table
#' @param source A name for external data source
-#' @param mode One of 'append', 'overwrite', 'error', 'ignore' save mode
+#' @param mode One of 'append', 'overwrite', 'error', 'ignore' save mode (it is 'error' by default)
#'
#' @family DataFrame functions
#' @rdname saveAsTable
@@ -1968,7 +1968,7 @@ setMethod("saveDF",
setMethod("saveAsTable",
signature(df = "DataFrame", tableName = "character", source = "character",
mode = "character"),
- function(df, tableName, source = NULL, mode="append", ...){
+ function(df, tableName, source = NULL, mode="error", ...){
if (is.null(source)) {
sqlContext <- get(".sparkRSQLsc", envir = .sparkREnv)
source <- callJMethod(sqlContext, "getConf", "spark.sql.sources.default",
http://git-wip-us.apache.org/repos/asf/spark/blob/2eb5af5f/docs/sparkr.md
----------------------------------------------------------------------
diff --git a/docs/sparkr.md b/docs/sparkr.md
index 0114878..9ddd2ed 100644
--- a/docs/sparkr.md
+++ b/docs/sparkr.md
@@ -148,7 +148,7 @@ printSchema(people)
</div>
The data sources API can also be used to save out DataFrames into multiple file formats. For example we can save the DataFrame from the previous example
-to a Parquet file using `write.df`
+to a Parquet file using `write.df` (Until Spark 1.6, the default mode for writes was `append`. It was changed in Spark 1.7 to `error` to match the Scala API)
<div data-lang="r" markdown="1">
{% highlight r %}
@@ -387,3 +387,10 @@ The following functions are masked by the SparkR package:
Since part of SparkR is modeled on the `dplyr` package, certain functions in SparkR share the same names with those in `dplyr`. Depending on the load order of the two packages, some functions from the package loaded first are masked by those in the package loaded after. In such case, prefix such calls with the package name, for instance, `SparkR::cume_dist(x)` or `dplyr::cume_dist(x)`.
You can inspect the search path in R with [`search()`](https://stat.ethz.ch/R-manual/R-devel/library/base/html/search.html)
+
+
+# Migration Guide
+
+## Upgrading From SparkR 1.6 to 1.7
+
+ - Until Spark 1.6, the default mode for writes was `append`. It was changed in Spark 1.7 to `error` to match the Scala API.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org