You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Hao Cheng <ha...@yahoo.com> on 2012/02/09 07:15:08 UTC

Delimiters for nested structures

Hi,

My data have some map of map structures with customized delimiters.

As per Hive documents, by default, '\001' is the field separator; starting from '\002', every 2 consecutive characters are the delimiters of 1 level. My data do not follow this rule in term of delimiters. I mostly just need to handle map of map. I do not find a way in create table statement to redefine delimiters for more than 1 level nested structures. I try not to transform the data as it was produced by some other upstream process. 

Any ideas on how to do that in Hive? Thank you for your helps!


Regards,
Hao

Re: Delimiters for nested structures

Posted by Mark Grover <mg...@oanda.com>.
That's correct. See this for reference: https://issues.apache.org/jira/browse/HIVE-365

Mark Grover, Business Intelligence Analyst
OANDA Corporation 

www: oanda.com www: fxtrade.com 
e: mgrover@oanda.com 

"Best Trading Platform" - World Finance's Forex Awards 2009. 
"The One to Watch" - Treasury Today's Adam Smith Awards 2009. 


----- Original Message -----
From: "Aniket Mokashi" <an...@gmail.com>
To: user@hive.apache.org, "Hao Cheng" <ha...@yahoo.com>
Sent: Thursday, February 9, 2012 2:34:29 AM
Subject: Re: Delimiters for nested structures

AFAIK, hive uses default delimiters nested data structures. There is no workaround for this for now. 
for ( int i = 3; i < serdeParams. separators . length ; i++) { 


serdeParams. separators [i] = ( byte ) (i + 1); 

} 

Thanks, 

Aniket 

On Wed, Feb 8, 2012 at 10:15 PM, Hao Cheng < haoc.cheng@yahoo.com > wrote: 





Hi, 


My data have some map of map structures with customized delimiters. 

As per Hive documents, by default, '\001' is the field separator; starting from '\002', every 2 consecutive characters are the delimiters of 1 level. My data do not follow this rule in term of delimiters. I mostly just need to handle map of map. I do not find a way in create table statement to redefine delimiters for more than 1 level nested structures. I try not to transform the data as it was produced by some other upstream process. 

Any ideas on how to do that in Hive? Thank you for your helps! 



Regards, 
Hao 



-- 
"...:::Aniket:::... Quetzalco@tl" 

Re: Delimiters for nested structures

Posted by Aniket Mokashi <an...@gmail.com>.
AFAIK, hive uses default delimiters nested data structures. There is no
workaround for this for now.
for (int i = 3; i < serdeParams.separators.length; i++) {

      serdeParams.separators[i] = (byte) (i + 1);

    }

Thanks,

Aniket

On Wed, Feb 8, 2012 at 10:15 PM, Hao Cheng <ha...@yahoo.com> wrote:

> Hi,
>
> My data have some map of map structures with customized delimiters.
> As per Hive documents, by default, '\001' is the field separator; starting
> from '\002', every 2 consecutive characters are the delimiters of 1 level.
> My data do not follow this rule in term of delimiters. I mostly just need
> to handle map of map. I do not find a way in create table statement to
> redefine delimiters for more than 1 level nested structures. I try not to
> transform the data as it was produced by some other upstream process.
> Any ideas on how to do that in Hive? Thank you for your helps!
>
> Regards,
> Hao
>



-- 
"...:::Aniket:::... Quetzalco@tl"

Re: Delimiters for nested structures

Posted by Evan Pollan <Ev...@bazaarvoice.com>.
+1. I've had good luck with json and get_json_object.

On Feb 9, 2012, at 7:39 AM, "Tucker, Matt" <Ma...@disney.com>> wrote:

What about creating a view that converts your data into JSON or XML?  You can then make use of the get_json_object<https://cwiki.apache.org/Hive/languagemanual-udf.html#LanguageManualUDF-getjsonobject>() or xpath<https://cwiki.apache.org/Hive/languagemanual-xpathudf.html>() functions.

Matt Tucker

From: Hao Cheng [mailto:haoc.cheng@yahoo.com]
Sent: Thursday, February 09, 2012 1:15 AM
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: Delimiters for nested structures

Hi,

My data have some map of map structures with customized delimiters.
As per Hive documents, by default, '\001' is the field separator; starting from '\002', every 2 consecutive characters are the delimiters of 1 level. My data do not follow this rule in term of delimiters. I mostly just need to handle map of map. I do not find a way in create table statement to redefine delimiters for more than 1 level nested structures. I try not to transform the data as it was produced by some other upstream process.
Any ideas on how to do that in Hive? Thank you for your helps!

Regards,
Hao

RE: Delimiters for nested structures

Posted by "Tucker, Matt" <Ma...@disney.com>.
What about creating a view that converts your data into JSON or XML?  You can then make use of the get_json_object<https://cwiki.apache.org/Hive/languagemanual-udf.html#LanguageManualUDF-getjsonobject>() or xpath<https://cwiki.apache.org/Hive/languagemanual-xpathudf.html>() functions.

Matt Tucker

From: Hao Cheng [mailto:haoc.cheng@yahoo.com]
Sent: Thursday, February 09, 2012 1:15 AM
To: user@hive.apache.org
Subject: Delimiters for nested structures

Hi,

My data have some map of map structures with customized delimiters.
As per Hive documents, by default, '\001' is the field separator; starting from '\002', every 2 consecutive characters are the delimiters of 1 level. My data do not follow this rule in term of delimiters. I mostly just need to handle map of map. I do not find a way in create table statement to redefine delimiters for more than 1 level nested structures. I try not to transform the data as it was produced by some other upstream process.
Any ideas on how to do that in Hive? Thank you for your helps!

Regards,
Hao