[jira] [Commented] (SPARK-11283) List column gets additional level of nesting when converted to Spark DataFrame

["Shivaram Venkataraman (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-11283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14974507#comment-14974507 ] 

Shivaram Venkataraman commented on SPARK-11283:
-----------------------------------------------

cc [~sunrui]

> List column gets additional level of nesting when converted to Spark DataFrame
> ------------------------------------------------------------------------------
>
>                 Key: SPARK-11283
>                 URL: https://issues.apache.org/jira/browse/SPARK-11283
>             Project: Spark
>          Issue Type: Bug
>          Components: SparkR
>    Affects Versions: 1.6.0
>         Environment: R 3.2.2, Spark build from master 487d409e71767c76399217a07af8de1bb0da7aa8
>            Reporter: Maciej Szymkiewicz
>
> When input data frame contains list column there is an additional level of nesting in a Spark DataFrame and as a result collected data is no longer identical to input:
> {code}
> ldf <- data.frame(row.names=1:2)
> ldf$x <- list(list(1), list(2))
> sdf <- createDataFrame(sqlContext, ldf)
> printSchema(sdf)
> ## root
> ##  |-- x: array (nullable = true)
> ##  |    |-- element: array (containsNull = true)
> ##  |    |    |-- element: double (containsNull = true)
> identical(ldf, collect(sdf))
> ## [1] FALSE
> {code}
> Comparing structure:
> Local df
> {code}
> unclass(ldf)
> ## $x
> ## $x[[1]]
> ## $x[[1]][[1]]
> ## [1] 1
> ##
> ## $x[[2]]
> ## $x[[2]][[1]]
> ## [1] 2
> ##
> ## attr(,"row.names")
> ## [1] 1 2
> {code}
> Collected
> {code}
> unclass(collect(sdf))
> ## $x
> ## $x[[1]]
> ## $x[[1]][[1]]
> ## $x[[1]][[1]][[1]]
> ## [1] 1
> ## 
> ## $x[[2]]
> ## $x[[2]][[1]]
> ## $x[[2]][[1]][[1]]
> ## [1] 2
> ##
> ## attr(,"row.names")
> ## [1] 1 2
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org