You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Mark Grover <mg...@oanda.com> on 2011/10/18 00:02:02 UTC
Hive Transform functionality using map datatype
Hi all,
Thank you in advance for your help.
I am using Hive Transform functionality to populate a table which has a column of Map<String, String> type.
Here is the syntax for the reduce part:
INSERT OVERWRITE TABLE
test_table
PARTITION (dt = '2011-01-31')
REDUCE
m.key, m.value
USING
'java -classpath test.jar com.comp.bi.Reducer'
AS
time, v1, v2;
Table test_table is defined to by doing:
create external table test_table (time INT, v1 STRING, v2 map<string, string>)
PARTITIONED BY (dt STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
STORED AS TEXTFILE;
When I run the query to populate test_table, I get an error:
FAILED: Error in semantic analysis: Cannot insert into target table because column number/types are different '2011-01-31': Cannot convert column from string to map<string,string>.
I have verified that the other columns match in number and type. If I convert the map<string, string> to be string type, the query doesn't error out.
Is it possible to populate a table with a complex type (map<string, string>) using streaming (i.e. transform)? If so, can any of you please suggest how to make the query succeed?
Thank you again!
Mark
Re: Hive Transform functionality using map datatype
Posted by Mark Grover <mg...@oanda.com>.
Thank you, Steve and Mark.
Both those links came in handy!
Here they are again for others' reference:
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Transform#LanguageManualTransform-TypingtheoutputofTRANSFORM
http://shmsoft.blogspot.com/2011/10/loading-inner-maps-in-hive.html
Mark
----- Original Message -----
From: "Steven Wong" <sw...@netflix.com>
To: user@hive.apache.org
Sent: Monday, October 17, 2011 9:19:22 PM
Subject: RE: Hive Transform functionality using map datatype
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Transform#LanguageManualTransform-TypingtheoutputofTRANSFORM
-----Original Message-----
From: Mark Grover [mailto:mgrover@oanda.com]
Sent: Monday, October 17, 2011 3:02 PM
To: user@hive.apache.org
Cc: Baiju Devani; Bob Tiernay; Denys Berestyuk
Subject: Hive Transform functionality using map datatype
Hi all,
Thank you in advance for your help.
I am using Hive Transform functionality to populate a table which has a column of Map<String, String> type.
Here is the syntax for the reduce part:
INSERT OVERWRITE TABLE
test_table
PARTITION (dt = '2011-01-31')
REDUCE
m.key, m.value
USING
'java -classpath test.jar com.comp.bi.Reducer'
AS
time, v1, v2;
Table test_table is defined to by doing:
create external table test_table (time INT, v1 STRING, v2 map<string, string>)
PARTITIONED BY (dt STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
STORED AS TEXTFILE;
When I run the query to populate test_table, I get an error:
FAILED: Error in semantic analysis: Cannot insert into target table because column number/types are different '2011-01-31': Cannot convert column from string to map<string,string>.
I have verified that the other columns match in number and type. If I convert the map<string, string> to be string type, the query doesn't error out.
Is it possible to populate a table with a complex type (map<string, string>) using streaming (i.e. transform)? If so, can any of you please suggest how to make the query succeed?
Thank you again!
Mark
RE: Hive Transform functionality using map datatype
Posted by Steven Wong <sw...@netflix.com>.
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Transform#LanguageManualTransform-TypingtheoutputofTRANSFORM
-----Original Message-----
From: Mark Grover [mailto:mgrover@oanda.com]
Sent: Monday, October 17, 2011 3:02 PM
To: user@hive.apache.org
Cc: Baiju Devani; Bob Tiernay; Denys Berestyuk
Subject: Hive Transform functionality using map datatype
Hi all,
Thank you in advance for your help.
I am using Hive Transform functionality to populate a table which has a column of Map<String, String> type.
Here is the syntax for the reduce part:
INSERT OVERWRITE TABLE
test_table
PARTITION (dt = '2011-01-31')
REDUCE
m.key, m.value
USING
'java -classpath test.jar com.comp.bi.Reducer'
AS
time, v1, v2;
Table test_table is defined to by doing:
create external table test_table (time INT, v1 STRING, v2 map<string, string>)
PARTITIONED BY (dt STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
STORED AS TEXTFILE;
When I run the query to populate test_table, I get an error:
FAILED: Error in semantic analysis: Cannot insert into target table because column number/types are different '2011-01-31': Cannot convert column from string to map<string,string>.
I have verified that the other columns match in number and type. If I convert the map<string, string> to be string type, the query doesn't error out.
Is it possible to populate a table with a complex type (map<string, string>) using streaming (i.e. transform)? If so, can any of you please suggest how to make the query succeed?
Thank you again!
Mark
Re: Hive Transform functionality using map datatype
Posted by Mark Kerzner <ma...@shmsoft.com>.
Mark,
see if this post on loading complex maps in Hive may be helpful,
http://shmsoft.blogspot.com/2011/10/loading-inner-maps-in-hive.html
Mark Kerzner
On Mon, Oct 17, 2011 at 5:02 PM, Mark Grover <mg...@oanda.com> wrote:
> Hi all,
> Thank you in advance for your help.
>
> I am using Hive Transform functionality to populate a table which has a
> column of Map<String, String> type.
>
> Here is the syntax for the reduce part:
>
> INSERT OVERWRITE TABLE
> test_table
> PARTITION (dt = '2011-01-31')
> REDUCE
> m.key, m.value
> USING
> 'java -classpath test.jar com.comp.bi.Reducer'
> AS
> time, v1, v2;
>
> Table test_table is defined to by doing:
> create external table test_table (time INT, v1 STRING, v2 map<string,
> string>)
> PARTITIONED BY (dt STRING)
> ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
> STORED AS TEXTFILE;
>
> When I run the query to populate test_table, I get an error:
> FAILED: Error in semantic analysis: Cannot insert into target table because
> column number/types are different '2011-01-31': Cannot convert column from
> string to map<string,string>.
>
> I have verified that the other columns match in number and type. If I
> convert the map<string, string> to be string type, the query doesn't error
> out.
>
> Is it possible to populate a table with a complex type (map<string,
> string>) using streaming (i.e. transform)? If so, can any of you please
> suggest how to make the query succeed?
>
> Thank you again!
>
> Mark
>
>
>
>
>
>