You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Pierre B <pi...@realimpactanalytics.com> on 2014/07/08 21:43:20 UTC
[Spark SQL]: Convert SchemaRDD back to RDD
Hi there!
1/ Is there a way to convert a SchemaRDD (for instance loaded from a parquet
file) back to a RDD of a given case class?
2/ Even better, is there a way to get the schema information from a
SchemaRDD ? I am trying to figure out how to properly get the various fields
of the Rows of a SchemaRDD. Knowing the schema (in the form of a Map?), I
guess I could nicely use getInt, getString, ..., on each row.
Parquet is really appealing for our project, for compression, columnar
access and embedded meta-data, but it would make much more sense if the
schema was available when loading.
Is there any plan to make this accessible?
Thanks
Pierre
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-Convert-SchemaRDD-back-to-RDD-tp9071.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: [Spark SQL]: Convert SchemaRDD back to RDD
Posted by Pierre B <pi...@realimpactanalytics.com>.
Cool Thanks Michael!
Message sent from a mobile device - excuse typos and abbreviations
> Le 8 juil. 2014 à 22:17, Michael Armbrust [via Apache Spark User List] <ml...@n3.nabble.com> a écrit :
>
>> On Tue, Jul 8, 2014 at 12:43 PM, Pierre B <[hidden email]> wrote:
>> 1/ Is there a way to convert a SchemaRDD (for instance loaded from a parquet
>> file) back to a RDD of a given case class?
>
> There may be someday, but doing so will either require a lot of reflection or a bunch of macro magic. So while I think this would be cool, it will probably be a while before we can implement it, and it'll likely be experimental.
>
>> 2/ Even better, is there a way to get the schema information from a
>> SchemaRDD ? I am trying to figure out how to properly get the various fields
>> of the Rows of a SchemaRDD. Knowing the schema (in the form of a Map?), I
>> guess I could nicely use getInt, getString, ..., on each row.
>
> We are actively working on this (SPARK-2179). Hopefully there will be a PR soon, and we are targeting the 1.1 release.
>
>
> If you reply to this email, your message will be added to the discussion below:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-Convert-SchemaRDD-back-to-RDD-tp9071p9084.html
> To unsubscribe from [Spark SQL]: Convert SchemaRDD back to RDD, click here.
> NAML
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-Convert-SchemaRDD-back-to-RDD-tp9071p9090.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: [Spark SQL]: Convert SchemaRDD back to RDD
Posted by Michael Armbrust <mi...@databricks.com>.
This is not currently supported. Right now you can only get RDD[Row] as
Ted suggested.
On Sun, Feb 22, 2015 at 2:52 PM, Ted Yu <yu...@gmail.com> wrote:
> Haven't found the method in
> http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.SchemaRDD
>
> The new DataFrame has this method:
>
> /**
> * Returns the content of the [[DataFrame]] as an [[RDD]] of [[Row]]s.
> * @group rdd
> */
> def rdd: RDD[Row] = {
>
> FYI
>
> On Sun, Feb 22, 2015 at 11:51 AM, stephane.collot <
> stephane.collot@gmail.com> wrote:
>
>> Hi Michael,
>>
>> I think that the feature (convert a SchemaRDD to a structured class RDD)
>> is
>> now available. But I didn't understand in the PR how exactly to do this.
>> Can
>> you give an example or doc links?
>>
>> Best regards
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-Convert-SchemaRDD-back-to-RDD-tp9071p21753.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
Re: [Spark SQL]: Convert SchemaRDD back to RDD
Posted by Ted Yu <yu...@gmail.com>.
Haven't found the method in
http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.SchemaRDD
The new DataFrame has this method:
/**
* Returns the content of the [[DataFrame]] as an [[RDD]] of [[Row]]s.
* @group rdd
*/
def rdd: RDD[Row] = {
FYI
On Sun, Feb 22, 2015 at 11:51 AM, stephane.collot <stephane.collot@gmail.com
> wrote:
> Hi Michael,
>
> I think that the feature (convert a SchemaRDD to a structured class RDD) is
> now available. But I didn't understand in the PR how exactly to do this.
> Can
> you give an example or doc links?
>
> Best regards
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-Convert-SchemaRDD-back-to-RDD-tp9071p21753.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: [Spark SQL]: Convert SchemaRDD back to RDD
Posted by "stephane.collot" <st...@gmail.com>.
Hi Michael,
I think that the feature (convert a SchemaRDD to a structured class RDD) is
now available. But I didn't understand in the PR how exactly to do this. Can
you give an example or doc links?
Best regards
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-Convert-SchemaRDD-back-to-RDD-tp9071p21753.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: [Spark SQL]: Convert SchemaRDD back to RDD
Posted by Michael Armbrust <mi...@databricks.com>.
On Tue, Jul 8, 2014 at 12:43 PM, Pierre B <
pierre.borckmans@realimpactanalytics.com> wrote:
>
> 1/ Is there a way to convert a SchemaRDD (for instance loaded from a
> parquet
> file) back to a RDD of a given case class?
>
There may be someday, but doing so will either require a lot of reflection
or a bunch of macro magic. So while I think this would be cool, it will
probably be a while before we can implement it, and it'll likely be
experimental.
> 2/ Even better, is there a way to get the schema information from a
> SchemaRDD ? I am trying to figure out how to properly get the various
> fields
> of the Rows of a SchemaRDD. Knowing the schema (in the form of a Map?), I
> guess I could nicely use getInt, getString, ..., on each row.
>
We are actively working on this (SPARK-2179
<https://issues.apache.org/jira/browse/SPARK-2179>). Hopefully there will
be a PR soon, and we are targeting the 1.1 release.