You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Robert McCarthy <ro...@gmail.com> on 2013/03/01 16:30:05 UTC
Re: jsonStorage and pig maps, not sure whats wrong with this?
Hi Dmitriy,
This looks like it was a bug in pig 10 within the JsonStorage class as the
same code now works in pig 11.
The script is is
A = load '/user/rob/events' USING PigStorage('\t') AS
(dt_dt:chararray,platform:chararray);
X = FOREACH A GENERATE dt_dt as timestamp,TOMAP('@tags',platform);
STORE X INTO '/user/rob/events.json' USING JsonStorage;
the data can be anything that has two columns that conform to chararray, it
doesn't matter.
On 27 February 2013 09:26, Dmitriy Ryaboy <dv...@gmail.com> wrote:
> Sounds odd. Can you send a complete script that reproduces the error
> (include sample data and load statements).
>
>
> On Thu, Feb 21, 2013 at 2:55 AM, Robert McCarthy <
> robert.mark.mccarthy@gmail.com> wrote:
>
> > If I have some information in A, that contains dt_dt and platform, I want
> > to store it in a different json format,
> > So I can create a simple new bag like this
> > X = FOREACH A GENERATE dt_dt as timestamp;
> > and it stores ok in JsonStorage
> > STORE X INTO '/user/rob/events.json' USING JsonStorage;
> >
> > I want to create a json object containing a pig map, because in the json
> > object I want "@tags": platform,
> > I can create the map ok like this..
> > X = FOREACH A GENERATE dt_dt as timestamp,TOMAP('@tags',platform);
> >
> > Describe and illustrate all show the pig data as expected.
> >
> > describe x
> > x: {timestamp: chararray,map[]}
> >
> > illustrate x
> > --------------------------------------------------
> > | x | timestamp:chararray | :map |
> > --------------------------------------------------
> > | | 2013-02-05 | {@tags=web} |
> > --------------------------------------------------
> >
> > but it will not store using JsonStorage
> > STORE X INTO '/user/rob/events.json' USING JsonStorage;
> >
> > Failed to parse: <line 1, column 15> mismatched input '[' expecting
> > IDENTIFIER_L
> >
> > I have tried without the @ to see if that was causing the error but get
> the
> > same result.
> > Any help much appreciated.
> >
> > Rob,
> >
>
Re: jsonStorage and pig maps, not sure whats wrong with this?
Posted by Robert McCarthy <ro...@gmail.com>.
Ah, pig 11 stores something but this seems to be generating an extra
"map_0" entry not the name value pair I was expecting.
I am getting
{"timestamp":"2013-01-31","map_0":{"@tags":"web"}}
{"timestamp":"2013-02-05","map_0":{"@tags":"web"}}
what I was hoping for was:
{"timestamp":"2013-01-31","@tags":"web"}
{"timestamp":"2013-02-05","@tags":"web"}
Anybody know how I can achieve this? I can't use
GENERATE platform as @tags
or
GENERATE platform as "@tags" or GENERATE platform as '@tags'
as it complains about the " ' @. Thought the TOMAP('@tags',platform) was a
clever work around.
On 1 March 2013 15:30, Robert McCarthy <ro...@gmail.com>wrote:
> Hi Dmitriy,
>
> This looks like it was a bug in pig 10 within the JsonStorage class as the
> same code now works in pig 11.
>
> The script is is
> A = load '/user/rob/events' USING PigStorage('\t') AS
> (dt_dt:chararray,platform:chararray);
> X = FOREACH A GENERATE dt_dt as timestamp,TOMAP('@tags',platform);
> STORE X INTO '/user/rob/events.json' USING JsonStorage;
>
> the data can be anything that has two columns that conform to chararray,
> it doesn't matter.
>
>
> On 27 February 2013 09:26, Dmitriy Ryaboy <dv...@gmail.com> wrote:
>
>> Sounds odd. Can you send a complete script that reproduces the error
>> (include sample data and load statements).
>>
>>
>> On Thu, Feb 21, 2013 at 2:55 AM, Robert McCarthy <
>> robert.mark.mccarthy@gmail.com> wrote:
>>
>> > If I have some information in A, that contains dt_dt and platform, I
>> want
>> > to store it in a different json format,
>> > So I can create a simple new bag like this
>> > X = FOREACH A GENERATE dt_dt as timestamp;
>> > and it stores ok in JsonStorage
>> > STORE X INTO '/user/rob/events.json' USING JsonStorage;
>> >
>> > I want to create a json object containing a pig map, because in the json
>> > object I want "@tags": platform,
>> > I can create the map ok like this..
>> > X = FOREACH A GENERATE dt_dt as timestamp,TOMAP('@tags',platform);
>> >
>> > Describe and illustrate all show the pig data as expected.
>> >
>> > describe x
>> > x: {timestamp: chararray,map[]}
>> >
>> > illustrate x
>> > --------------------------------------------------
>> > | x | timestamp:chararray | :map |
>> > --------------------------------------------------
>> > | | 2013-02-05 | {@tags=web} |
>> > --------------------------------------------------
>> >
>> > but it will not store using JsonStorage
>> > STORE X INTO '/user/rob/events.json' USING JsonStorage;
>> >
>> > Failed to parse: <line 1, column 15> mismatched input '[' expecting
>> > IDENTIFIER_L
>> >
>> > I have tried without the @ to see if that was causing the error but get
>> the
>> > same result.
>> > Any help much appreciated.
>> >
>> > Rob,
>> >
>>
>
>