You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jerry Lam <ch...@gmail.com> on 2015/09/26 01:21:42 UTC

Spark SQL: Native Support for LATERAL VIEW EXPLODE

Hi sparkers,

Anyone knows how to do LATERAL VIEW EXPLODE without HiveContext?
I don't want to start up a metastore and derby just because I need LATERAL
VIEW EXPLODE.

I have been trying but I always get the exception like this:

Name: java.lang.RuntimeException
Message: [1.68] failure: ``union'' expected but identifier view found

with the query look like:

"select items from purhcases lateral view explode(purchase_items) tbl as
items"

Best Regards,

Jerry

Re: Spark SQL: Native Support for LATERAL VIEW EXPLODE

Posted by Michael Armbrust <mi...@databricks.com>.
No, you would just have to do another select to pull out the fields you are
interested in.

On Sat, Sep 26, 2015 at 11:11 AM, Jerry Lam <ch...@gmail.com> wrote:

> Hi Michael,
>
> Thanks for the tip. With dataframe, is it possible to explode some
> selected fields in each purchase_items?
> Since purchase_items is an array of item and each item has a number of
> fields (for example product_id and price), is it possible to just explode
> these two fields directly using dataframe?
>
> Best Regards,
>
>
> Jerry
>
> On Fri, Sep 25, 2015 at 7:53 PM, Michael Armbrust <mi...@databricks.com>
> wrote:
>
>> The SQL parser without HiveContext is really simple, which is why I
>> generally recommend users use HiveContext.  However, you can do it with
>> dataframes:
>>
>> import org.apache.spark.sql.functions._
>> table("purchases").select(explode(df("purchase_items")).as("item"))
>>
>>
>>
>> On Fri, Sep 25, 2015 at 4:21 PM, Jerry Lam <ch...@gmail.com> wrote:
>>
>>> Hi sparkers,
>>>
>>> Anyone knows how to do LATERAL VIEW EXPLODE without HiveContext?
>>> I don't want to start up a metastore and derby just because I need
>>> LATERAL VIEW EXPLODE.
>>>
>>> I have been trying but I always get the exception like this:
>>>
>>> Name: java.lang.RuntimeException
>>> Message: [1.68] failure: ``union'' expected but identifier view found
>>>
>>> with the query look like:
>>>
>>> "select items from purhcases lateral view explode(purchase_items) tbl as
>>> items"
>>>
>>> Best Regards,
>>>
>>> Jerry
>>>
>>>
>>
>

Re: Spark SQL: Native Support for LATERAL VIEW EXPLODE

Posted by Jerry Lam <ch...@gmail.com>.
Hi Michael,

Thanks for the tip. With dataframe, is it possible to explode some selected
fields in each purchase_items?
Since purchase_items is an array of item and each item has a number of
fields (for example product_id and price), is it possible to just explode
these two fields directly using dataframe?

Best Regards,


Jerry

On Fri, Sep 25, 2015 at 7:53 PM, Michael Armbrust <mi...@databricks.com>
wrote:

> The SQL parser without HiveContext is really simple, which is why I
> generally recommend users use HiveContext.  However, you can do it with
> dataframes:
>
> import org.apache.spark.sql.functions._
> table("purchases").select(explode(df("purchase_items")).as("item"))
>
>
>
> On Fri, Sep 25, 2015 at 4:21 PM, Jerry Lam <ch...@gmail.com> wrote:
>
>> Hi sparkers,
>>
>> Anyone knows how to do LATERAL VIEW EXPLODE without HiveContext?
>> I don't want to start up a metastore and derby just because I need
>> LATERAL VIEW EXPLODE.
>>
>> I have been trying but I always get the exception like this:
>>
>> Name: java.lang.RuntimeException
>> Message: [1.68] failure: ``union'' expected but identifier view found
>>
>> with the query look like:
>>
>> "select items from purhcases lateral view explode(purchase_items) tbl as
>> items"
>>
>> Best Regards,
>>
>> Jerry
>>
>>
>

Re: Spark SQL: Native Support for LATERAL VIEW EXPLODE

Posted by Michael Armbrust <mi...@databricks.com>.
The SQL parser without HiveContext is really simple, which is why I
generally recommend users use HiveContext.  However, you can do it with
dataframes:

import org.apache.spark.sql.functions._
table("purchases").select(explode(df("purchase_items")).as("item"))



On Fri, Sep 25, 2015 at 4:21 PM, Jerry Lam <ch...@gmail.com> wrote:

> Hi sparkers,
>
> Anyone knows how to do LATERAL VIEW EXPLODE without HiveContext?
> I don't want to start up a metastore and derby just because I need LATERAL
> VIEW EXPLODE.
>
> I have been trying but I always get the exception like this:
>
> Name: java.lang.RuntimeException
> Message: [1.68] failure: ``union'' expected but identifier view found
>
> with the query look like:
>
> "select items from purhcases lateral view explode(purchase_items) tbl as
> items"
>
> Best Regards,
>
> Jerry
>
>