You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Renato Marroquín Mogrovejo <re...@gmail.com> on 2015/03/15 15:22:53 UTC

[Spark SQL]: Convert JavaSchemaRDD back to JavaRDD of a specific class

Hi Spark experts,

Is there a way to convert a JavaSchemaRDD (for instance loaded from a
parquet file) back to a JavaRDD of a given case class? I read on
stackOverFlow[1] that I could do a select over the parquet file and then by
reflection get the fields out, but I guess that would be an overkill.
Then I saw [2] from 2014 which says that this feature would be available in
the future. So could you please let me know how I can accomplish this?
Thanks in advance!


Renato M.

[1]
http://stackoverflow.com/questions/26181353/how-to-convert-spark-schemardd-into-rdd-of-my-case-class
[2]
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-Convert-SchemaRDD-back-to-RDD-td9071.html

Re: [Spark SQL]: Convert JavaSchemaRDD back to JavaRDD of a specific class

Posted by Cheng Lian <li...@gmail.com>.
Currently there’s no convenient way to convert a 
|SchemaRDD|/|JavaSchemaRDD| back to an |RDD|/|JavaRDD| of some case 
class. But you can convert an |RDD|/|JavaRDD| into an 
|RDD[Row]|/|JavaRDD<Row>| using |schemaRdd.rdd| and |new 
JavaRDD<Row>(schemaRdd.rdd)|.

Cheng

On 3/15/15 10:22 PM, Renato Marroquín Mogrovejo wrote:

> Hi Spark experts,
>
> Is there a way to convert a JavaSchemaRDD (for instance loaded from a 
> parquet file) back to a JavaRDD of a given case class? I read on 
> stackOverFlow[1] that I could do a select over the parquet file and 
> then by reflection get the fields out, but I guess that would be an 
> overkill.
> Then I saw [2] from 2014 which says that this feature would be 
> available in the future. So could you please let me know how I can 
> accomplish this? Thanks in advance!
>
>
> Renato M.
>
> [1] 
> http://stackoverflow.com/questions/26181353/how-to-convert-spark-schemardd-into-rdd-of-my-case-class
> [2] 
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-Convert-SchemaRDD-back-to-RDD-td9071.html

​