You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Clark Fitzgerald (JIRA)" <ji...@apache.org> on 2016/07/28 22:51:20 UTC
[jira] [Comment Edited] (SPARK-16785) dapply doesn't return array
or raw columns
[ https://issues.apache.org/jira/browse/SPARK-16785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15398312#comment-15398312 ]
Clark Fitzgerald edited comment on SPARK-16785 at 7/28/16 10:51 PM:
--------------------------------------------------------------------
[~shivaram] and I have had some email correspondence on this:
bq. parallelizing data containing binary values: To create a SparkDataFrame from a local R data frame we first convert it to a list of rows and then partition the rows. This happens on the driver 1. The rows are then reconstructed back into a data.frame on the workers 2. The rbind.data.frame fails for the the binary example. I've created a small reproducible example at https://gist.github.com/shivaram/2f07208a8ebc0832098328fa0a3fac9d
1 https://github.com/apache/spark/blob/0869b3a5f028b64c2da511e70b02ab42f65fc949/R/pkg/R/SQLContext.R#L209
2 https://github.com/apache/spark/blob/0869b3a5f028b64c2da511e70b02ab42f65fc949/R/pkg/inst/worker/worker.R#L39
was (Author: clarkfitzg):
[~shivaram] and I have had some email correspondence on this:
bq. parallelizing data containing binary values: To create a SparkDataFrame from a local R data frame we first convert it to a list of rows and then partition the rows. This happens on the driver 1. The rows are then reconstructed back into a data.frame on the workers 2. The rbind.data.frame fails for the the binary example. I've created a small reproducible example at https://gist.github.com/shivaram/2f07208a8ebc0832098328fa0a3fac9d
1 https://github.com/apache/spark/blob/0869b3a5f028b64c2da511e70b02ab42f65fc949/R/pkg/R/SQLContext.R#L209
2 https://github.com/apache/spark/blob/0869b3a5f028b64c2da511e70b02ab42f65fc949/R/pkg/inst/worker/worker.R#L39
To fix this I propose to treat the rows as a list of dataframes instead of as a list of lists:
{{rowlist = split(df, 1:nrow(df))}}
{{df2 = do.call(rbind, rowlist)}}
> dapply doesn't return array or raw columns
> ------------------------------------------
>
> Key: SPARK-16785
> URL: https://issues.apache.org/jira/browse/SPARK-16785
> Project: Spark
> Issue Type: Bug
> Components: SparkR
> Affects Versions: 2.0.0
> Environment: Mac OS X
> Reporter: Clark Fitzgerald
> Priority: Minor
>
> Calling SparkR::dapplyCollect with R functions that return dataframes produces an error. This comes up when returning columns of binary data- ie. serialized fitted models. Also happens when functions return columns containing vectors.
> The error message:
> R computation failed with
> Error in (function (..., deparse.level = 1, make.row.names = TRUE, stringsAsFactors = default.stringsAsFactors()) :
> invalid list argument: all variables should have the same length
> Reproducible example: https://github.com/clarkfitzg/phd_research/blob/master/ddR/spark/sparkR_dapplyCollect7.R
> Relates to SPARK-16611
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org