You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Jeff Storey <st...@gmail.com> on 2014/03/21 13:38:03 UTC
Load Hive query result with array field into pig
I'm executing a hive query in which one of the fields an array and writing
it to a file using:
INSERT OVERWRITE '/path/to/output' SELECT ...
This query works well. I would like to load this data into pig, but I'm
quite sure how to get the array properly into pig.
My output file from the query doesn't look like it has any array notation
when using the default ^A separator.
Is there a way I can execute a query with an array field output that can
then be directly loaded into pig as a tuple?
Thank you.
Re: Load Hive query result with array field into pig
Posted by Jeff Storey <st...@gmail.com>.
Sorry for another post on this thread. I had an error in my pigscript that
had the wrong unicode character to split on. Using STRSPLIT worked well.
On Fri, Mar 21, 2014 at 8:46 AM, Jeff Storey <st...@gmail.com> wrote:
> Correction - it looks like the query uses \u002 to separate array elements
> and \u001 to separate the other fields. The question is still similar
> though in wondering how I can load that array into pig.
>
> Note - If my data is formatted as a tsv with parentheses surrounding the
> array:
>
> (element1,element2,element3) anotherfield
>
> This loads properly when I use LOAD '/my/tsvfile' USING PigStorage('\t')
> AS (elements:tuple(),afield:chararray);
>
>
> On Fri, Mar 21, 2014 at 8:38 AM, Jeff Storey <st...@gmail.com>wrote:
>
>> I'm executing a hive query in which one of the fields an array and
>> writing it to a file using:
>>
>> INSERT OVERWRITE '/path/to/output' SELECT ...
>>
>> This query works well. I would like to load this data into pig, but I'm
>> quite sure how to get the array properly into pig.
>>
>> My output file from the query doesn't look like it has any array notation
>> when using the default ^A separator.
>>
>> Is there a way I can execute a query with an array field output that can
>> then be directly loaded into pig as a tuple?
>>
>> Thank you.
>>
>
>
Re: Load Hive query result with array field into pig
Posted by Jeff Storey <st...@gmail.com>.
Correction - it looks like the query uses \u002 to separate array elements
and \u001 to separate the other fields. The question is still similar
though in wondering how I can load that array into pig.
Note - If my data is formatted as a tsv with parentheses surrounding the
array:
(element1,element2,element3) anotherfield
This loads properly when I use LOAD '/my/tsvfile' USING PigStorage('\t') AS
(elements:tuple(),afield:chararray);
On Fri, Mar 21, 2014 at 8:38 AM, Jeff Storey <st...@gmail.com> wrote:
> I'm executing a hive query in which one of the fields an array and writing
> it to a file using:
>
> INSERT OVERWRITE '/path/to/output' SELECT ...
>
> This query works well. I would like to load this data into pig, but I'm
> quite sure how to get the array properly into pig.
>
> My output file from the query doesn't look like it has any array notation
> when using the default ^A separator.
>
> Is there a way I can execute a query with an array field output that can
> then be directly loaded into pig as a tuple?
>
> Thank you.
>