You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Mark Kerzner <ma...@shmsoft.com> on 2011/10/06 00:34:02 UTC
Best way to import complex data into Hive?
Hi,
I have some transactional data, which has a few map elements, but one more
complex - array of maps. Something like this
CREATE EXTERNAL TABLE mytable
(
log_timestamp string,
price map <string,string>,
opt array <map<string, string>>
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\001'
COLLECTION ITEMS TERMINATED BY '\004'
MAP KEYS TERMINATED BY '\002'
STORED AS TextFile;
However, the data is a bit more complex, and I also use '\003' and '\005\'
for delimiters, only I have no place to tell Hive about it.
So it seems that I can define data structures like opt array <map<string,
string>>, but I don't know how to import such data.
What would be the advice?
Thank you. Sincerely,
Mark
Re: Best way to import complex data into Hive?
Posted by Mark Kerzner <ma...@shmsoft.com>.
Thank you, Steven,
that was so good that I had to put it on my blog,
http://shmsoft.blogspot.com/2011/10/loading-inner-maps-in-hive.html
Mark
On Wed, Oct 5, 2011 at 8:12 PM, Steven Wong <sw...@netflix.com> wrote:
>
> http://mail-archives.apache.org/mod_mbox/hadoop-hive-user/201009.mbox/%3C4F6B25AFFFCAFE44B6259A412D5F9B101C07A390@ExchMBX104.netflix.com%3Eis a similar situation. Basically you have to use the default delimiters.
> ****
>
> ** **
>
> ** **
>
> *From:* Mark Kerzner [mailto:mark.kerzner@shmsoft.com]
> *Sent:* Wednesday, October 05, 2011 3:34 PM
> *To:* Hive user
> *Subject:* Best way to import complex data into Hive?****
>
> ** **
>
> Hi,
>
> I have some transactional data, which has a few map elements, but one more
> complex - array of maps. Something like this
>
> CREATE EXTERNAL TABLE mytable
> (
> log_timestamp string,
> price map <string,string>,
> opt array <map<string, string>>
> )
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY '\001'
> COLLECTION ITEMS TERMINATED BY '\004'
> MAP KEYS TERMINATED BY '\002'
> STORED AS TextFile;
>
> However, the data is a bit more complex, and I also use '\003' and '\005\'
> for delimiters, only I have no place to tell Hive about it.
>
> So it seems that I can define data structures like opt array <map<string,
> string>>, but I don't know how to import such data.
>
> What would be the advice?
>
> Thank you. Sincerely,
> Mark****
>
Re: Best way to import complex data into Hive?
Posted by Mark Kerzner <ma...@shmsoft.com>.
Wow, Steve,
who could have thought?! That's hidden and undocumented, but useful. I guess
"read the code, stupid" applies to me in this case.
Thank you,
Mark
On Wed, Oct 5, 2011 at 8:12 PM, Steven Wong <sw...@netflix.com> wrote:
>
> http://mail-archives.apache.org/mod_mbox/hadoop-hive-user/201009.mbox/%3C4F6B25AFFFCAFE44B6259A412D5F9B101C07A390@ExchMBX104.netflix.com%3Eis a similar situation. Basically you have to use the default delimiters.
> ****
>
> ** **
>
> ** **
>
> *From:* Mark Kerzner [mailto:mark.kerzner@shmsoft.com]
> *Sent:* Wednesday, October 05, 2011 3:34 PM
> *To:* Hive user
> *Subject:* Best way to import complex data into Hive?****
>
> ** **
>
> Hi,
>
> I have some transactional data, which has a few map elements, but one more
> complex - array of maps. Something like this
>
> CREATE EXTERNAL TABLE mytable
> (
> log_timestamp string,
> price map <string,string>,
> opt array <map<string, string>>
> )
> ROW FORMAT DELIMITED
> FIELDS TERMINATED BY '\001'
> COLLECTION ITEMS TERMINATED BY '\004'
> MAP KEYS TERMINATED BY '\002'
> STORED AS TextFile;
>
> However, the data is a bit more complex, and I also use '\003' and '\005\'
> for delimiters, only I have no place to tell Hive about it.
>
> So it seems that I can define data structures like opt array <map<string,
> string>>, but I don't know how to import such data.
>
> What would be the advice?
>
> Thank you. Sincerely,
> Mark****
>
RE: Best way to import complex data into Hive?
Posted by Steven Wong <sw...@netflix.com>.
http://mail-archives.apache.org/mod_mbox/hadoop-hive-user/201009.mbox/%3C4F6B25AFFFCAFE44B6259A412D5F9B101C07A390@ExchMBX104.netflix.com%3E is a similar situation. Basically you have to use the default delimiters.
From: Mark Kerzner [mailto:mark.kerzner@shmsoft.com]
Sent: Wednesday, October 05, 2011 3:34 PM
To: Hive user
Subject: Best way to import complex data into Hive?
Hi,
I have some transactional data, which has a few map elements, but one more complex - array of maps. Something like this
CREATE EXTERNAL TABLE mytable
(
log_timestamp string,
price map <string,string>,
opt array <map<string, string>>
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\001'
COLLECTION ITEMS TERMINATED BY '\004'
MAP KEYS TERMINATED BY '\002'
STORED AS TextFile;
However, the data is a bit more complex, and I also use '\003' and '\005\' for delimiters, only I have no place to tell Hive about it.
So it seems that I can define data structures like opt array <map<string, string>>, but I don't know how to import such data.
What would be the advice?
Thank you. Sincerely,
Mark