You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Akshat Aranya <aa...@gmail.com> on 2014/10/17 18:30:04 UTC

Attaching schema to RDD created from Parquet file

Hi,

How can I convert an RDD loaded from a Parquet file into its original type:

case class Person(name: String, age: Int)

val rdd: RDD[Person] = ...
rdd.saveAsParquetFile("people")

val rdd2: sqlContext.parquetFile("people")

How can I map rdd2 back into an RDD[Person]?  All of the examples just show
how to use the RDD loaded from Parquet using SQL.

Re: Attaching schema to RDD created from Parquet file

Posted by Michael Armbrust <mi...@databricks.com>.

This has to be done manually:

rdd2.map(row => Person(row.getString(0), row.getInt(1)))

On Fri, Oct 17, 2014 at 9:30 AM, Akshat Aranya <aa...@gmail.com> wrote:

> Hi,
>
> How can I convert an RDD loaded from a Parquet file into its original type:
>
> case class Person(name: String, age: Int)
>
> val rdd: RDD[Person] = ...
> rdd.saveAsParquetFile("people")
>
> val rdd2: sqlContext.parquetFile("people")
>
> How can I map rdd2 back into an RDD[Person]?  All of the examples just
> show how to use the RDD loaded from Parquet using SQL.
>