[GitHub] [spark] felixcheung commented on a change in pull request #23969: [SPARK-26920][R] Deduplicate type checking across Arrow optimization in SparkR

[GitBox <gi...@apache.org>]

felixcheung commented on a change in pull request #23969: [SPARK-26920][R] Deduplicate type checking across Arrow optimization in SparkR
URL: https://github.com/apache/spark/pull/23969#discussion_r263989081
 
 

 ##########
 File path: R/pkg/R/SQLContext.R
 ##########
 @@ -197,17 +197,40 @@ writeToFileInArrow <- function(fileName, rdf, numPartitions) {
   }
 }
 
-checkTypeRequirementForArrow <- function(dataHead, schema) {
-  # Currenty Arrow optimization does not support raw for now.
-  # Also, it does not support explicit float type set by users. It leads to
-  # incorrect conversion. We will fall back to the path without Arrow optimization.
-  if (any(sapply(dataHead, is.raw))) {
-    stop("Arrow optimization with R DataFrame does not support raw type yet.")
-  }
-  if (inherits(schema, "structType")) {
-    if (any(sapply(schema$fields(), function(x) x$dataType.toString() == "FloatType"))) {
-      stop("Arrow optimization with R DataFrame does not support FloatType type yet.")
+getSchema <- function(schema, firstRow = NULL, rdd = NULL) {
 
 Review comment:
   I think that's fine

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org