You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Haitao Yao <ya...@gmail.com> on 2012/02/06 08:12:54 UTC

How to load customized map data schema

Hi, all
	out data format for map is Key:Value|Key:Value , how can I load the data into map type?  Can pig define the map delimiter like hive? 
	thanks.

Re: How to load customized map data schema

Posted by Daniel Dai <da...@hortonworks.com>.
data = load '/tmp/titan_mixi_prod.10' using he.HEStorage() as
(uid:long, payload:map[], ts:long, type:int);

Daniel

2012/2/6 Haitao Yao <ya...@gmail.com>:
> I've wrote my own loader and here's the error:
> grunt>data = load '/tmp/titan_mixi_prod.10' using he.HEStorage() as (uid:long, payload:map, ts:long, type:int);
>
> ERROR 1200: <line 1, column 84>  mismatched input ',' expecting LEFT_BRACKET
>
> Failed to parse: <line 1, column 84>  mismatched input ',' expecting LEFT_BRACKET
>        at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:222)
>        at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:164)
>        at org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1609)
>        at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582)
>        at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
>        at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:942)
>        at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386)
>        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
>        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
>        at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
>        at org.apache.pig.Main.run(Main.java:495)
>        at org.apache.pig.Main.main(Main.java:111)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>
>
> it seems like the error came before my loader is invoked.
>
> How can i solve this?
>
>
> 在 2012-2-6，下午4:14， Daniel Dai 写道：
>
>> Currently PigStorage only takes:
>> [key#value,key#value]
>> You will have to preprocess it before using PigStorage. Sure you can
>> also write your own loader.
>>
>> Daniel
>>
>> On Sun, Feb 5, 2012 at 11:12 PM, Haitao Yao <ya...@gmail.com> wrote:
>>> Hi, all
>>>        out data format for map is Key:Value|Key:Value , how can I load the data into map type?  Can pig define the map delimiter like hive?
>>>        thanks.
>>>
>>>
>>>
>

Re: How to load customized map data schema

Posted by Haitao Yao <ya...@gmail.com>.
I've wrote my own loader and here's the error:
grunt>data = load '/tmp/titan_mixi_prod.10' using he.HEStorage() as (uid:long, payload:map, ts:long, type:int);

ERROR 1200: <line 1, column 84>  mismatched input ',' expecting LEFT_BRACKET

Failed to parse: <line 1, column 84>  mismatched input ',' expecting LEFT_BRACKET
        at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:222)
        at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:164)
        at org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1609)
        at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582)
        at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
        at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:942)
        at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
        at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
        at org.apache.pig.Main.run(Main.java:495)
        at org.apache.pig.Main.main(Main.java:111)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)


it seems like the error came before my loader is invoked. 

How can i solve this?


在 2012-2-6，下午4:14， Daniel Dai 写道：

> Currently PigStorage only takes:
> [key#value,key#value]
> You will have to preprocess it before using PigStorage. Sure you can
> also write your own loader.
> 
> Daniel
> 
> On Sun, Feb 5, 2012 at 11:12 PM, Haitao Yao <ya...@gmail.com> wrote:
>> Hi, all
>>        out data format for map is Key:Value|Key:Value , how can I load the data into map type?  Can pig define the map delimiter like hive?
>>        thanks.
>> 
>> 
>>

Re: How to load customized map data schema

Posted by Daniel Dai <da...@hortonworks.com>.
Currently PigStorage only takes:
[key#value,key#value]
You will have to preprocess it before using PigStorage. Sure you can
also write your own loader.

Daniel

On Sun, Feb 5, 2012 at 11:12 PM, Haitao Yao <ya...@gmail.com> wrote:
> Hi, all
>        out data format for map is Key:Value|Key:Value , how can I load the data into map type?  Can pig define the map delimiter like hive?
>        thanks.
>
>
>