[jira] [Resolved] (SPARK-7160) Support converting DataFrames to typed RDDs.

["Michael Armbrust (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-7160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael Armbrust resolved SPARK-7160.
-------------------------------------
       Resolution: Fixed
    Fix Version/s: 1.6.0

As I commented on the PR, I think the goals of this JIRA are going to be accomplished by the Dataset API.  Since we are getting close to 1.6 I'm going to close this as fixed, but if there are tests or things from the existing PR that we can port over I think that would be great.  Also, if there is anything missing from the implementation in master I'd love to know about it.

> Support converting DataFrames to typed RDDs.
> --------------------------------------------
>
>                 Key: SPARK-7160
>                 URL: https://issues.apache.org/jira/browse/SPARK-7160
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL
>    Affects Versions: 1.3.1
>            Reporter: Ray Ortigas
>            Assignee: Ray Ortigas
>            Priority: Critical
>             Fix For: 1.6.0
>
>
> As a Spark user still working with RDDs, I'd like the ability to convert a DataFrame to a typed RDD.
> For example, if I've converted RDDs to DataFrames so that I could save them as Parquet or CSV files, I would like to rebuild the RDD from those files automatically rather than writing the row-to-type conversion myself.
> {code}
> val rdd0 = sc.parallelize(Seq(Food("apple", 1), Food("banana", 2), Food("cherry", 3)))
> val df0 = rdd0.toDF()
> df0.save("foods.parquet")
> val df1 = sqlContext.load("foods.parquet")
> val rdd1 = df1.toTypedRDD[Food]()
> // rdd0 and rdd1 should have the same elements
> {code}
> I originally submitted a smaller PR for spark-csv <https://github.com/databricks/spark-csv/pull/52>, but Reynold Xin suggested that converting a DataFrame to a typed RDD wasn't something specific to spark-csv.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org