You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jaonary Rabarisoa <ja...@gmail.com> on 2015/02/25 00:06:56 UTC

[ML][SQL] Select UserDefinedType attribute in a DataFrame

Hi all,

I have a DataFrame that contains a user defined type. The type is an image
with the following attribute

*class Image(w: Int, h: Int, data: Vector)*

In my DataFrame, images are stored in column named "image" that corresponds
to the following case class

*case class LabeledImage(label: Int, image: Image)*

How can I select image.data attribute of my image object and view it as a
column of a DataFrame ?

I'd like to do something like

*val featureDF = imagesDF.select("image.data").as("features")*


Cheers,

Jao

Re: [ML][SQL] Select UserDefinedType attribute in a DataFrame

Posted by Jaonary Rabarisoa <ja...@gmail.com>.
By doing so, I got the following error :

Exception in thread "main" org.apache.spark.sql.AnalysisException: GetField
is not valid on fields

Seems that it doesn't like image.data expression.

On Wed, Feb 25, 2015 at 12:37 AM, Xiangrui Meng <me...@gmail.com> wrote:

> Btw, the correct syntax for alias should be
> `df.select($"image.data".as("features"))`.
>
> On Tue, Feb 24, 2015 at 3:35 PM, Xiangrui Meng <me...@gmail.com> wrote:
> > If you make `Image` a case class, then select("image.data") should work.
> >
> > On Tue, Feb 24, 2015 at 3:06 PM, Jaonary Rabarisoa <ja...@gmail.com>
> wrote:
> >> Hi all,
> >>
> >> I have a DataFrame that contains a user defined type. The type is an
> image
> >> with the following attribute
> >>
> >> class Image(w: Int, h: Int, data: Vector)
> >>
> >> In my DataFrame, images are stored in column named "image" that
> corresponds
> >> to the following case class
> >>
> >> case class LabeledImage(label: Int, image: Image)
> >>
> >> How can I select image.data attribute of my image object and view it as
> a
> >> column of a DataFrame ?
> >>
> >> I'd like to do something like
> >>
> >> val featureDF = imagesDF.select("image.data").as("features")
> >>
> >>
> >> Cheers,
> >>
> >> Jao
>

Re: [ML][SQL] Select UserDefinedType attribute in a DataFrame

Posted by Xiangrui Meng <me...@gmail.com>.
Btw, the correct syntax for alias should be
`df.select($"image.data".as("features"))`.

On Tue, Feb 24, 2015 at 3:35 PM, Xiangrui Meng <me...@gmail.com> wrote:
> If you make `Image` a case class, then select("image.data") should work.
>
> On Tue, Feb 24, 2015 at 3:06 PM, Jaonary Rabarisoa <ja...@gmail.com> wrote:
>> Hi all,
>>
>> I have a DataFrame that contains a user defined type. The type is an image
>> with the following attribute
>>
>> class Image(w: Int, h: Int, data: Vector)
>>
>> In my DataFrame, images are stored in column named "image" that corresponds
>> to the following case class
>>
>> case class LabeledImage(label: Int, image: Image)
>>
>> How can I select image.data attribute of my image object and view it as a
>> column of a DataFrame ?
>>
>> I'd like to do something like
>>
>> val featureDF = imagesDF.select("image.data").as("features")
>>
>>
>> Cheers,
>>
>> Jao

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: [ML][SQL] Select UserDefinedType attribute in a DataFrame

Posted by Xiangrui Meng <me...@gmail.com>.
If you make `Image` a case class, then select("image.data") should work.

On Tue, Feb 24, 2015 at 3:06 PM, Jaonary Rabarisoa <ja...@gmail.com> wrote:
> Hi all,
>
> I have a DataFrame that contains a user defined type. The type is an image
> with the following attribute
>
> class Image(w: Int, h: Int, data: Vector)
>
> In my DataFrame, images are stored in column named "image" that corresponds
> to the following case class
>
> case class LabeledImage(label: Int, image: Image)
>
> How can I select image.data attribute of my image object and view it as a
> column of a DataFrame ?
>
> I'd like to do something like
>
> val featureDF = imagesDF.select("image.data").as("features")
>
>
> Cheers,
>
> Jao

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org