You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by felixcheung <gi...@git.apache.org> on 2015/10/22 08:55:29 UTC
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
GitHub user felixcheung opened a pull request:
https://github.com/apache/spark/pull/9218
[SPARK-9319][SPARKR] Add support for setting column names, types
Add support for for colnames, colnames<-, coltypes<-
I will merge with PR 8984 (coltypes) once it is in, possibly looking into mapping R type names.
@shivaram @sun-rui
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/felixcheung/spark colnamescoltypes
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9218.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9218
----
commit 071f29f998f86a5a05744c703bf6a9a2384c3805
Author: felixcheung <fe...@hotmail.com>
Date: 2015-10-22T06:48:43Z
Add support for colnames, colnames<-, coltypes<-
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-151379530
@sun-rui That's a great point, `coltypes()` as its signature is defined, would only return a list of simple types.
But how would one create a DataFrame with complex type from R? I tried a bit and couldn't get it to work. Either I get `Unsupported type for DataFrame: factor` or `unexpected type: environment`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/9218#discussion_r42899259
--- Diff: R/pkg/R/DataFrame.R ---
@@ -276,6 +276,57 @@ setMethod("names<-",
}
})
+#' @rdname columns
+#' @name colnames
+setMethod("colnames",
+ signature(x = "DataFrame"),
+ function(x) {
+ columns(x)
+ })
+
+#' @rdname columns
+#' @name colnames<-
+setMethod("colnames<-",
+ signature(x = "DataFrame", value = "character"),
+ function(x, value) {
+ sdf <- callJMethod(x@sdf, "toDF", as.list(value))
+ dataFrame(sdf)
+ })
+
+#' coltypes
+#'
+#' Set the column types of a DataFrame.
+#'
+#' @name coltypes
+#' @param x (DataFrame)
+#' @return value (character) A character vector with the target column types for the given DataFrame
+#' @rdname coltypes
+#' @aliases coltypes
+#' @export
+#' @examples
+#'\dontrun{
+#' sc <- sparkR.init()
+#' sqlContext <- sparkRSQL.init(sc)
+#' path <- "path/to/file.json"
+#' df <- jsonFile(sqlContext, path)
+#' coltypes(df) <- c("string", "integer")
+#'}
+setMethod("coltypes<-",
--- End diff --
That's correct. I'm hoping #8984 can be merged soon so I could add a new reverse mapping in the same place. I could make this [WIP] if you'd like
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-152868688
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/9218#discussion_r42785428
--- Diff: R/pkg/R/DataFrame.R ---
@@ -276,6 +276,57 @@ setMethod("names<-",
}
})
+#' @rdname columns
+#' @name colnames
--- End diff --
Although it is simply wrapping another method, having example and some description helps with in-line documentation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by sun-rui <gi...@git.apache.org>.
Github user sun-rui commented on a diff in the pull request:
https://github.com/apache/spark/pull/9218#discussion_r43983636
--- Diff: R/pkg/R/DataFrame.R ---
@@ -276,6 +276,75 @@ setMethod("names<-",
}
})
+#' @rdname columns
+#' @name colnames
+setMethod("colnames",
+ signature(x = "DataFrame"),
+ function(x) {
+ columns(x)
+ })
+
+#' @rdname columns
+#' @name colnames<-
+setMethod("colnames<-",
+ signature(x = "DataFrame", value = "character"),
+ function(x, value) {
+ sdf <- callJMethod(x@sdf, "toDF", as.list(value))
+ dataFrame(sdf)
+ })
+
+rToScalaTypes <- new.env()
+rToScalaTypes[["integer"]] <- "integer" # in R, integer is 32bit
+rToScalaTypes[["numeric"]] <- "double" # in R, numeric == double which is 64bit
+rToScalaTypes[["double"]] <- "double"
+rToScalaTypes[["character"]] <- "string"
+rToScalaTypes[["logical"]] <- "boolean"
+
+#' coltypes
+#'
+#' Set the column types of a DataFrame.
+#'
+#' @name coltypes
+#' @param x (DataFrame)
+#' @return value (character) A character vector with the target column types for the given
+#' DataFrame. Column types can be one of integer, numeric/double, character, logical, or NA
+#' to keep that column as-is.
+#' @rdname coltypes
+#' @aliases coltypes
+#' @export
+#' @examples
+#'\dontrun{
+#' sc <- sparkR.init()
+#' sqlContext <- sparkRSQL.init(sc)
+#' path <- "path/to/file.json"
+#' df <- jsonFile(sqlContext, path)
+#' coltypes(df) <- c("character", "integer")
+#' coltypes(df) <- c(NA, "numeric")
+#'}
+setMethod("coltypes<-",
+ signature(x = "DataFrame", value = "character"),
+ function(x, value) {
+ cols <- columns(x)
+ ncols <- length(cols)
+ if (length(value) == 0 || length(value) != ncols) {
--- End diff --
it is good for code robustness, and we may support emptyDataFrame later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-150126791
**[Test build #44137 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44137/consoleFull)** for PR 9218 at commit [`071f29f`](https://github.com/apache/spark/commit/071f29f998f86a5a05744c703bf6a9a2384c3805).
* This patch **fails some tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-152884866
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/9218#discussion_r42801664
--- Diff: R/pkg/R/DataFrame.R ---
@@ -276,6 +276,57 @@ setMethod("names<-",
}
})
+#' @rdname columns
+#' @name colnames
+setMethod("colnames",
+ signature(x = "DataFrame"),
+ function(x) {
+ columns(x)
+ })
+
+#' @rdname columns
+#' @name colnames<-
+setMethod("colnames<-",
+ signature(x = "DataFrame", value = "character"),
+ function(x, value) {
+ sdf <- callJMethod(x@sdf, "toDF", as.list(value))
+ dataFrame(sdf)
+ })
+
+#' coltypes
+#'
+#' Set the column types of a DataFrame.
+#'
+#' @name coltypes
+#' @param x (DataFrame)
+#' @return value (character) A character vector with the target column types for the given DataFrame
+#' @rdname coltypes
+#' @aliases coltypes
+#' @export
+#' @examples
+#'\dontrun{
+#' sc <- sparkR.init()
+#' sqlContext <- sparkRSQL.init(sc)
+#' path <- "path/to/file.json"
+#' df <- jsonFile(sqlContext, path)
+#' coltypes(df) <- c("string", "integer")
+#'}
+setMethod("coltypes<-",
--- End diff --
Certainly, it is in PR 8984 by @olarayej
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-156017477
**[Test build #45718 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45718/consoleFull)** for PR 9218 at commit [`8730a37`](https://github.com/apache/spark/commit/8730a375c59aa470e066cb2ce50c1d599e816898).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/9218#discussion_r43698343
--- Diff: R/pkg/R/DataFrame.R ---
@@ -276,6 +276,75 @@ setMethod("names<-",
}
})
+#' @rdname columns
+#' @name colnames
+setMethod("colnames",
+ signature(x = "DataFrame"),
+ function(x) {
+ columns(x)
+ })
+
+#' @rdname columns
+#' @name colnames<-
+setMethod("colnames<-",
+ signature(x = "DataFrame", value = "character"),
+ function(x, value) {
+ sdf <- callJMethod(x@sdf, "toDF", as.list(value))
+ dataFrame(sdf)
+ })
+
+rToScalaTypes <- new.env()
+rToScalaTypes[["integer"]] <- "integer" # in R, integer is 32bit
+rToScalaTypes[["numeric"]] <- "double" # in R, numeric == double which is 64bit
+rToScalaTypes[["double"]] <- "double"
+rToScalaTypes[["character"]] <- "string"
+rToScalaTypes[["logical"]] <- "boolean"
+
+#' coltypes
+#'
+#' Set the column types of a DataFrame.
+#'
+#' @name coltypes
+#' @param x (DataFrame)
+#' @return value (character) A character vector with the target column types for the given
+#' DataFrame. Column types can be one of integer, numeric/double, character, logical, or NA
+#' to keep that column as-is.
+#' @rdname coltypes
+#' @aliases coltypes
+#' @export
+#' @examples
+#'\dontrun{
+#' sc <- sparkR.init()
+#' sqlContext <- sparkRSQL.init(sc)
+#' path <- "path/to/file.json"
+#' df <- jsonFile(sqlContext, path)
+#' coltypes(df) <- c("character", "integer")
+#' coltypes(df) <- c(NA, "numeric")
+#'}
+setMethod("coltypes<-",
+ signature(x = "DataFrame", value = "character"),
+ function(x, value) {
+ cols <- columns(x)
+ ncols <- length(cols)
+ if (length(value) == 0 || length(value) != ncols) {
--- End diff --
It could do that. I would rather be concise even if only a few more lines because
- it's already checking for length == 0
- not sure we could create a DataFrame with 0 col from R
- no point to set column data type on a DataFrame having no column
Thought?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-156020838
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-152868672
**[Test build #44769 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44769/consoleFull)** for PR 9218 at commit [`49f7117`](https://github.com/apache/spark/commit/49f71179726a70ce129ea1284cc83bd113f594f8).
* This patch **fails SparkR unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by sun-rui <gi...@git.apache.org>.
Github user sun-rui commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-151399054
@felixcheung, type inferring works for complex types in createDataFrame(). You can refer to the test case for "create DataFrame with complex types" in test_sparkSQL.R.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/9218#discussion_r42884395
--- Diff: R/pkg/R/DataFrame.R ---
@@ -276,6 +276,57 @@ setMethod("names<-",
}
})
+#' @rdname columns
+#' @name colnames
+setMethod("colnames",
+ signature(x = "DataFrame"),
+ function(x) {
+ columns(x)
+ })
+
+#' @rdname columns
+#' @name colnames<-
+setMethod("colnames<-",
+ signature(x = "DataFrame", value = "character"),
+ function(x, value) {
+ sdf <- callJMethod(x@sdf, "toDF", as.list(value))
+ dataFrame(sdf)
+ })
+
+#' coltypes
+#'
+#' Set the column types of a DataFrame.
+#'
+#' @name coltypes
+#' @param x (DataFrame)
+#' @return value (character) A character vector with the target column types for the given DataFrame
+#' @rdname coltypes
+#' @aliases coltypes
+#' @export
+#' @examples
+#'\dontrun{
+#' sc <- sparkR.init()
+#' sqlContext <- sparkRSQL.init(sc)
+#' path <- "path/to/file.json"
+#' df <- jsonFile(sqlContext, path)
+#' coltypes(df) <- c("string", "integer")
+#'}
+setMethod("coltypes<-",
--- End diff --
So this is a little tricky. In #8984 we are converting the SparkSQL types to R types. So in that case for consistency we should take in R types here (i.e character, numeric etc.) and convert them to SparkSQL types
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-152882211
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/9218#discussion_r43694097
--- Diff: R/pkg/R/DataFrame.R ---
@@ -276,6 +276,75 @@ setMethod("names<-",
}
})
+#' @rdname columns
+#' @name colnames
+setMethod("colnames",
+ signature(x = "DataFrame"),
+ function(x) {
+ columns(x)
+ })
+
+#' @rdname columns
+#' @name colnames<-
+setMethod("colnames<-",
+ signature(x = "DataFrame", value = "character"),
+ function(x, value) {
+ sdf <- callJMethod(x@sdf, "toDF", as.list(value))
+ dataFrame(sdf)
+ })
+
+rToScalaTypes <- new.env()
+rToScalaTypes[["integer"]] <- "integer" # in R, integer is 32bit
+rToScalaTypes[["numeric"]] <- "double" # in R, numeric == double which is 64bit
+rToScalaTypes[["double"]] <- "double"
+rToScalaTypes[["character"]] <- "string"
+rToScalaTypes[["logical"]] <- "boolean"
--- End diff --
yes, that's the plan.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-152774672
Any more thought on this and #8984? It looks like we are cutting the 1.6.0 release very soon.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-152774865
My take is that its even fine if we only support basic types through `coltypes<-` for now and throw a `complex types not supported` error message for now. That way the API gets established and we can make it richer across releases.
@sun-rui @olarayej What are your thoughts on getting the API in time for 1.6 ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-152862908
```
Error : /home/jenkins/workspace/SparkPullRequestBuilder/R/pkg/man/colnames.Rd: Sections \title, and \name must exist and be unique in Rd files
ERROR: installing Rd objects failed for package 'SparkR'
```
This is odd, there shouldn't be a Rd file for colnames, I haven't changed this and it worked before
```
#' @rdname columns
#' @name colnames
setMethod("colnames",
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-156020677
**[Test build #45718 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45718/consoleFull)** for PR 9218 at commit [`8730a37`](https://github.com/apache/spark/commit/8730a375c59aa470e066cb2ce50c1d599e816898).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-156020841
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45718/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by sun-rui <gi...@git.apache.org>.
Github user sun-rui commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-150809533
I am not clear that for both coltypes() and coltypes<-(), how to represent complex types in R types? do you have idea?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-150281730
cc @olarayej
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-152861491
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by sun-rui <gi...@git.apache.org>.
Github user sun-rui commented on a diff in the pull request:
https://github.com/apache/spark/pull/9218#discussion_r43604658
--- Diff: R/pkg/R/DataFrame.R ---
@@ -276,6 +276,75 @@ setMethod("names<-",
}
})
+#' @rdname columns
+#' @name colnames
+setMethod("colnames",
+ signature(x = "DataFrame"),
+ function(x) {
+ columns(x)
+ })
+
+#' @rdname columns
+#' @name colnames<-
+setMethod("colnames<-",
+ signature(x = "DataFrame", value = "character"),
+ function(x, value) {
+ sdf <- callJMethod(x@sdf, "toDF", as.list(value))
+ dataFrame(sdf)
+ })
+
+rToScalaTypes <- new.env()
+rToScalaTypes[["integer"]] <- "integer" # in R, integer is 32bit
+rToScalaTypes[["numeric"]] <- "double" # in R, numeric == double which is 64bit
+rToScalaTypes[["double"]] <- "double"
+rToScalaTypes[["character"]] <- "string"
+rToScalaTypes[["logical"]] <- "boolean"
+
+#' coltypes
--- End diff --
coltypes<- ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by sun-rui <gi...@git.apache.org>.
Github user sun-rui commented on a diff in the pull request:
https://github.com/apache/spark/pull/9218#discussion_r43604767
--- Diff: R/pkg/R/DataFrame.R ---
@@ -276,6 +276,75 @@ setMethod("names<-",
}
})
+#' @rdname columns
+#' @name colnames
+setMethod("colnames",
+ signature(x = "DataFrame"),
+ function(x) {
+ columns(x)
+ })
+
+#' @rdname columns
+#' @name colnames<-
+setMethod("colnames<-",
+ signature(x = "DataFrame", value = "character"),
+ function(x, value) {
+ sdf <- callJMethod(x@sdf, "toDF", as.list(value))
+ dataFrame(sdf)
+ })
+
+rToScalaTypes <- new.env()
+rToScalaTypes[["integer"]] <- "integer" # in R, integer is 32bit
+rToScalaTypes[["numeric"]] <- "double" # in R, numeric == double which is 64bit
+rToScalaTypes[["double"]] <- "double"
+rToScalaTypes[["character"]] <- "string"
+rToScalaTypes[["logical"]] <- "boolean"
+
+#' coltypes
+#'
+#' Set the column types of a DataFrame.
+#'
+#' @name coltypes
+#' @param x (DataFrame)
+#' @return value (character) A character vector with the target column types for the given
+#' DataFrame. Column types can be one of integer, numeric/double, character, logical, or NA
+#' to keep that column as-is.
+#' @rdname coltypes
+#' @aliases coltypes
+#' @export
+#' @examples
+#'\dontrun{
+#' sc <- sparkR.init()
+#' sqlContext <- sparkRSQL.init(sc)
+#' path <- "path/to/file.json"
+#' df <- jsonFile(sqlContext, path)
+#' coltypes(df) <- c("character", "integer")
+#' coltypes(df) <- c(NA, "numeric")
+#'}
+setMethod("coltypes<-",
+ signature(x = "DataFrame", value = "character"),
+ function(x, value) {
+ cols <- columns(x)
+ ncols <- length(cols)
+ if (length(value) == 0 || length(value) != ncols) {
--- End diff --
It is possible that a DataFrame has 0 column.
So propose the code:
```
if (length(value) != ncols) {
stop("Length of type vector should match the number of columns for DataFrame")
}
if(ncols <= 0) {
return(x)
}
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by sun-rui <gi...@git.apache.org>.
Github user sun-rui commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-150164199
could we support both names() and colnames()?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-152862537
**[Test build #44767 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44767/consoleFull)** for PR 9218 at commit [`fae9bba`](https://github.com/apache/spark/commit/fae9bbaede31ce3ff6326c1a2cbd12c52b3243d9).
* This patch **fails some tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-152862543
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-150126758
**[Test build #44137 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44137/consoleFull)** for PR 9218 at commit [`071f29f`](https://github.com/apache/spark/commit/071f29f998f86a5a05744c703bf6a9a2384c3805).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung closed the pull request at:
https://github.com/apache/spark/pull/9218
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-156016213
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by falaki <gi...@git.apache.org>.
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/9218#discussion_r42785718
--- Diff: R/pkg/R/DataFrame.R ---
@@ -276,6 +276,57 @@ setMethod("names<-",
}
})
+#' @rdname columns
+#' @name colnames
+setMethod("colnames",
+ signature(x = "DataFrame"),
+ function(x) {
+ columns(x)
+ })
+
+#' @rdname columns
+#' @name colnames<-
+setMethod("colnames<-",
+ signature(x = "DataFrame", value = "character"),
+ function(x, value) {
+ sdf <- callJMethod(x@sdf, "toDF", as.list(value))
+ dataFrame(sdf)
+ })
+
+#' coltypes
+#'
+#' Set the column types of a DataFrame.
+#'
+#' @name coltypes
+#' @param x (DataFrame)
+#' @return value (character) A character vector with the target column types for the given DataFrame
+#' @rdname coltypes
+#' @aliases coltypes
+#' @export
+#' @examples
+#'\dontrun{
+#' sc <- sparkR.init()
+#' sqlContext <- sparkRSQL.init(sc)
+#' path <- "path/to/file.json"
+#' df <- jsonFile(sqlContext, path)
+#' coltypes(df) <- c("string", "integer")
+#'}
+setMethod("coltypes<-",
--- End diff --
Do you think users may expect ```coltypes``` in addition to ```coltypes<-```? I think having it would be nice.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-152882214
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/9218#discussion_r42786136
--- Diff: R/pkg/R/DataFrame.R ---
@@ -276,6 +276,57 @@ setMethod("names<-",
}
})
+#' @rdname columns
+#' @name colnames
--- End diff --
In R doc, it will be included under the `columns` page because of `@rdname` notation. https://cran.r-project.org/web/packages/roxygen2/vignettes/rd.html
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-152867650
**[Test build #44769 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44769/consoleFull)** for PR 9218 at commit [`49f7117`](https://github.com/apache/spark/commit/49f71179726a70ce129ea1284cc83bd113f594f8).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by sun-rui <gi...@git.apache.org>.
Github user sun-rui commented on a diff in the pull request:
https://github.com/apache/spark/pull/9218#discussion_r43604632
--- Diff: R/pkg/R/DataFrame.R ---
@@ -276,6 +276,75 @@ setMethod("names<-",
}
})
+#' @rdname columns
+#' @name colnames
+setMethod("colnames",
+ signature(x = "DataFrame"),
+ function(x) {
+ columns(x)
+ })
+
+#' @rdname columns
+#' @name colnames<-
+setMethod("colnames<-",
+ signature(x = "DataFrame", value = "character"),
+ function(x, value) {
+ sdf <- callJMethod(x@sdf, "toDF", as.list(value))
+ dataFrame(sdf)
+ })
+
+rToScalaTypes <- new.env()
+rToScalaTypes[["integer"]] <- "integer" # in R, integer is 32bit
+rToScalaTypes[["numeric"]] <- "double" # in R, numeric == double which is 64bit
+rToScalaTypes[["double"]] <- "double"
+rToScalaTypes[["character"]] <- "string"
+rToScalaTypes[["logical"]] <- "boolean"
--- End diff --
We can keep R types -> SQLTypes mapping here, but it would be better we can refactor mapping between R types and SQLTypes later (to combine with SQLTypes -> R types mapping in "colTypes()")
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-150125099
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/9218#discussion_r43694062
--- Diff: R/pkg/R/DataFrame.R ---
@@ -276,6 +276,75 @@ setMethod("names<-",
}
})
+#' @rdname columns
+#' @name colnames
+setMethod("colnames",
+ signature(x = "DataFrame"),
+ function(x) {
+ columns(x)
+ })
+
+#' @rdname columns
+#' @name colnames<-
+setMethod("colnames<-",
+ signature(x = "DataFrame", value = "character"),
+ function(x, value) {
+ sdf <- callJMethod(x@sdf, "toDF", as.list(value))
--- End diff --
I tried actually but couldn't get that to work - it ended up calling names<- of data.frame.
Since this is one line I thought it's ok this way.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-152884821
**[Test build #44783 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44783/consoleFull)** for PR 9218 at commit [`8d3f043`](https://github.com/apache/spark/commit/8d3f04375a6010164f3efef28fa1105ce59eea16).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by sun-rui <gi...@git.apache.org>.
Github user sun-rui commented on a diff in the pull request:
https://github.com/apache/spark/pull/9218#discussion_r43717935
--- Diff: R/pkg/R/DataFrame.R ---
@@ -276,6 +276,75 @@ setMethod("names<-",
}
})
+#' @rdname columns
+#' @name colnames
+setMethod("colnames",
+ signature(x = "DataFrame"),
+ function(x) {
+ columns(x)
+ })
+
+#' @rdname columns
+#' @name colnames<-
+setMethod("colnames<-",
+ signature(x = "DataFrame", value = "character"),
+ function(x, value) {
+ sdf <- callJMethod(x@sdf, "toDF", as.list(value))
+ dataFrame(sdf)
+ })
+
+rToScalaTypes <- new.env()
+rToScalaTypes[["integer"]] <- "integer" # in R, integer is 32bit
+rToScalaTypes[["numeric"]] <- "double" # in R, numeric == double which is 64bit
+rToScalaTypes[["double"]] <- "double"
+rToScalaTypes[["character"]] <- "string"
+rToScalaTypes[["logical"]] <- "boolean"
+
+#' coltypes
+#'
+#' Set the column types of a DataFrame.
+#'
+#' @name coltypes
+#' @param x (DataFrame)
+#' @return value (character) A character vector with the target column types for the given
+#' DataFrame. Column types can be one of integer, numeric/double, character, logical, or NA
+#' to keep that column as-is.
+#' @rdname coltypes
+#' @aliases coltypes
+#' @export
+#' @examples
+#'\dontrun{
+#' sc <- sparkR.init()
+#' sqlContext <- sparkRSQL.init(sc)
+#' path <- "path/to/file.json"
+#' df <- jsonFile(sqlContext, path)
+#' coltypes(df) <- c("character", "integer")
+#' coltypes(df) <- c(NA, "numeric")
+#'}
+setMethod("coltypes<-",
+ signature(x = "DataFrame", value = "character"),
+ function(x, value) {
+ cols <- columns(x)
+ ncols <- length(cols)
+ if (length(value) == 0 || length(value) != ncols) {
--- End diff --
Since SQLContext.emptyDataFrame allows DataFrames with 0 column, we'd better not exclude the possibility.
Yes, your code checks length == 0, for suppose the following call:
```
coltypes(df) <- character()
```
Your code will throw error message:
"Length of type vector should match the number of columns for DataFrame"
But I think this case can be silently ignored.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by sun-rui <gi...@git.apache.org>.
Github user sun-rui commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-152797513
yes, I agree: throw a complex types not supported error message
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-150126793
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44137/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-152861497
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-150126792
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-152865405
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-152884867
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44783/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-152883770
**[Test build #44783 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44783/consoleFull)** for PR 9218 at commit [`8d3f043`](https://github.com/apache/spark/commit/8d3f04375a6010164f3efef28fa1105ce59eea16).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by sun-rui <gi...@git.apache.org>.
Github user sun-rui commented on a diff in the pull request:
https://github.com/apache/spark/pull/9218#discussion_r43717683
--- Diff: R/pkg/R/DataFrame.R ---
@@ -276,6 +276,75 @@ setMethod("names<-",
}
})
+#' @rdname columns
+#' @name colnames
+setMethod("colnames",
+ signature(x = "DataFrame"),
+ function(x) {
+ columns(x)
+ })
+
+#' @rdname columns
+#' @name colnames<-
+setMethod("colnames<-",
+ signature(x = "DataFrame", value = "character"),
+ function(x, value) {
+ sdf <- callJMethod(x@sdf, "toDF", as.list(value))
+ dataFrame(sdf)
+ })
+
+rToScalaTypes <- new.env()
+rToScalaTypes[["integer"]] <- "integer" # in R, integer is 32bit
+rToScalaTypes[["numeric"]] <- "double" # in R, numeric == double which is 64bit
+rToScalaTypes[["double"]] <- "double"
+rToScalaTypes[["character"]] <- "string"
+rToScalaTypes[["logical"]] <- "boolean"
+
+#' coltypes
--- End diff --
Looks at the "names" and "names<-" in DataFrame.R, seems "coltypes" and "coltypes<-" should share a same function description. We should co-operate between this PR and the PR for "coltypes"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/9218#discussion_r43698022
--- Diff: R/pkg/R/DataFrame.R ---
@@ -276,6 +276,75 @@ setMethod("names<-",
}
})
+#' @rdname columns
+#' @name colnames
+setMethod("colnames",
+ signature(x = "DataFrame"),
+ function(x) {
+ columns(x)
+ })
+
+#' @rdname columns
+#' @name colnames<-
+setMethod("colnames<-",
+ signature(x = "DataFrame", value = "character"),
+ function(x, value) {
+ sdf <- callJMethod(x@sdf, "toDF", as.list(value))
+ dataFrame(sdf)
+ })
+
+rToScalaTypes <- new.env()
+rToScalaTypes[["integer"]] <- "integer" # in R, integer is 32bit
+rToScalaTypes[["numeric"]] <- "double" # in R, numeric == double which is 64bit
+rToScalaTypes[["double"]] <- "double"
+rToScalaTypes[["character"]] <- "string"
+rToScalaTypes[["logical"]] <- "boolean"
+
+#' coltypes
--- End diff --
R doc is usually under the getter name instead of the setter (`coltypes` vs `coltypes<-`)
Also it seems `devtools::document(pkg="./pkg", roclets=c("rd"))` doesn't know how to handle `<-` in the name.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by sun-rui <gi...@git.apache.org>.
Github user sun-rui commented on a diff in the pull request:
https://github.com/apache/spark/pull/9218#discussion_r43603571
--- Diff: R/pkg/R/DataFrame.R ---
@@ -276,6 +276,75 @@ setMethod("names<-",
}
})
+#' @rdname columns
+#' @name colnames
+setMethod("colnames",
+ signature(x = "DataFrame"),
+ function(x) {
+ columns(x)
+ })
+
+#' @rdname columns
+#' @name colnames<-
+setMethod("colnames<-",
+ signature(x = "DataFrame", value = "character"),
+ function(x, value) {
+ sdf <- callJMethod(x@sdf, "toDF", as.list(value))
--- End diff --
Is it possible to directly call names<-()?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-152865413
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-152868689
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44769/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by sun-rui <gi...@git.apache.org>.
Github user sun-rui commented on a diff in the pull request:
https://github.com/apache/spark/pull/9218#discussion_r43717596
--- Diff: R/pkg/R/DataFrame.R ---
@@ -276,6 +276,75 @@ setMethod("names<-",
}
})
+#' @rdname columns
+#' @name colnames
+setMethod("colnames",
+ signature(x = "DataFrame"),
+ function(x) {
+ columns(x)
+ })
+
+#' @rdname columns
+#' @name colnames<-
+setMethod("colnames<-",
+ signature(x = "DataFrame", value = "character"),
+ function(x, value) {
+ sdf <- callJMethod(x@sdf, "toDF", as.list(value))
--- End diff --
I did experiment and found the following code works:
```
setMethod("colnames<-",
signature(x = "DataFrame", value = "character"),
function(x, value) {
names(x) <- value
x
})
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-152862485
**[Test build #44767 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44767/consoleFull)** for PR 9218 at commit [`fae9bba`](https://github.com/apache/spark/commit/fae9bbaede31ce3ff6326c1a2cbd12c52b3243d9).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/9218#discussion_r43718136
--- Diff: R/pkg/R/DataFrame.R ---
@@ -276,6 +276,75 @@ setMethod("names<-",
}
})
+#' @rdname columns
+#' @name colnames
+setMethod("colnames",
+ signature(x = "DataFrame"),
+ function(x) {
+ columns(x)
+ })
+
+#' @rdname columns
+#' @name colnames<-
+setMethod("colnames<-",
+ signature(x = "DataFrame", value = "character"),
+ function(x, value) {
+ sdf <- callJMethod(x@sdf, "toDF", as.list(value))
+ dataFrame(sdf)
+ })
+
+rToScalaTypes <- new.env()
+rToScalaTypes[["integer"]] <- "integer" # in R, integer is 32bit
+rToScalaTypes[["numeric"]] <- "double" # in R, numeric == double which is 64bit
+rToScalaTypes[["double"]] <- "double"
+rToScalaTypes[["character"]] <- "string"
+rToScalaTypes[["logical"]] <- "boolean"
+
+#' coltypes
--- End diff --
that's the plan too. could we merge #8984 now? :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/9218#discussion_r43718270
--- Diff: R/pkg/R/DataFrame.R ---
@@ -276,6 +276,75 @@ setMethod("names<-",
}
})
+#' @rdname columns
+#' @name colnames
+setMethod("colnames",
+ signature(x = "DataFrame"),
+ function(x) {
+ columns(x)
+ })
+
+#' @rdname columns
+#' @name colnames<-
+setMethod("colnames<-",
+ signature(x = "DataFrame", value = "character"),
+ function(x, value) {
+ sdf <- callJMethod(x@sdf, "toDF", as.list(value))
+ dataFrame(sdf)
+ })
+
+rToScalaTypes <- new.env()
+rToScalaTypes[["integer"]] <- "integer" # in R, integer is 32bit
+rToScalaTypes[["numeric"]] <- "double" # in R, numeric == double which is 64bit
+rToScalaTypes[["double"]] <- "double"
+rToScalaTypes[["character"]] <- "string"
+rToScalaTypes[["logical"]] <- "boolean"
+
+#' coltypes
+#'
+#' Set the column types of a DataFrame.
+#'
+#' @name coltypes
+#' @param x (DataFrame)
+#' @return value (character) A character vector with the target column types for the given
+#' DataFrame. Column types can be one of integer, numeric/double, character, logical, or NA
+#' to keep that column as-is.
+#' @rdname coltypes
+#' @aliases coltypes
+#' @export
+#' @examples
+#'\dontrun{
+#' sc <- sparkR.init()
+#' sqlContext <- sparkRSQL.init(sc)
+#' path <- "path/to/file.json"
+#' df <- jsonFile(sqlContext, path)
+#' coltypes(df) <- c("character", "integer")
+#' coltypes(df) <- c(NA, "numeric")
+#'}
+setMethod("coltypes<-",
+ signature(x = "DataFrame", value = "character"),
+ function(x, value) {
+ cols <- columns(x)
+ ncols <- length(cols)
+ if (length(value) == 0 || length(value) != ncols) {
--- End diff --
I agree, that's why we should check for length = 0. I am not sure about supporting it though since `emptyDataFrame` is not callable from R, AFAIK.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-150125119
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-150311963
@sun-rui `names` and `names<-` are already there, this is to add `colnames`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-156016257
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9218#issuecomment-152862545
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44767/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org