You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Cesar Flores <ce...@gmail.com> on 2015/03/06 16:22:16 UTC
Data Frame types
The SchemaRDD supports the storage of user defined classes. However, in
order to do that, the user class needs to extends the UserDefinedType interface
(see for example VectorUDT in org.apache.spark.mllib.linalg).
My question is: Do the new Data Frame Structure (to be released in spark
1.3) will be able to handle user defined classes too? Do user classes will
need to extend they will need to define the same approach?
--
Cesar Flores
Re: Data Frame types
Posted by Michael Armbrust <mi...@databricks.com>.
No, the UDT API is not a public API as we have not stabilized the
implementation. For this reason its only accessible to projects inside of
Spark.
On Fri, Mar 6, 2015 at 8:25 AM, Jaonary Rabarisoa <ja...@gmail.com> wrote:
> Hi Cesar,
>
> Yes, you can define an UDT with the new DataFrame, the same way that
> SchemaRDD did.
>
> Jaonary
>
> On Fri, Mar 6, 2015 at 4:22 PM, Cesar Flores <ce...@gmail.com> wrote:
>
>>
>> The SchemaRDD supports the storage of user defined classes. However, in
>> order to do that, the user class needs to extends the UserDefinedType interface
>> (see for example VectorUDT in org.apache.spark.mllib.linalg).
>>
>> My question is: Do the new Data Frame Structure (to be released in spark
>> 1.3) will be able to handle user defined classes too? Do user classes will
>> need to extend they will need to define the same approach?
>>
>>
>> --
>> Cesar Flores
>>
>
>
Re: Data Frame types
Posted by Jaonary Rabarisoa <ja...@gmail.com>.
Hi Cesar,
Yes, you can define an UDT with the new DataFrame, the same way that
SchemaRDD did.
Jaonary
On Fri, Mar 6, 2015 at 4:22 PM, Cesar Flores <ce...@gmail.com> wrote:
>
> The SchemaRDD supports the storage of user defined classes. However, in
> order to do that, the user class needs to extends the UserDefinedType interface
> (see for example VectorUDT in org.apache.spark.mllib.linalg).
>
> My question is: Do the new Data Frame Structure (to be released in spark
> 1.3) will be able to handle user defined classes too? Do user classes will
> need to extend they will need to define the same approach?
>
>
> --
> Cesar Flores
>