You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Dan Young <da...@gmail.com> on 2012/03/25 06:35:58 UTC
Trying to store a bag of tuples using AvroStorage.
Hello all,
I'm trying to store a bag of tuples using AvroStorage but am not able to
figure out what I'm doing wrong (or if it' supported). What I have is the
following:
grunt>illustrate c;
....
....
---------------------------------------------------------------------------------------------
| c | franchise_id:int |
cast_and_crew:bag{:tuple(full_name:chararray)}|
---------------------------------------------------------------------------------------------
| | 213939 | {(Wang Junzheng), (Li Ling)}
|
---------------------------------------------------------------------------------------------
When I try to store is via AvroStorage I'm getting an error. I've tried
both with and without specifying a schema I've created a gist and posted
it here with the Pig Stack Trace.....
https://gist.github.com/2191363
If anyone has insight into what I'm doing wrong, and can help, I would
greatly appreciate the guidance.
Regards,
Dan
Re: Trying to store a bag of tuples using AvroStorage.
Posted by Dan Young <da...@gmail.com>.
Dooohh....thank you for pointing that out....I thought I ran that thru
jsonlint.....That seemed to fix it........
Regards,
Dano
On Tue, Apr 3, 2012 at 12:11 PM, Bill Graham <bi...@gmail.com> wrote:
> In the schema approach the error is that your json is invalid. You're
> missing a second '}' before the last ']'.
>
>
>
> On Tue, Apr 3, 2012 at 10:32 AM, Dan Young <da...@gmail.com> wrote:
>
> > I just updated my pig from svn repo and now am using the latest from
> trunk:
> >
> > pig -i
> > Apache Pig version 0.11.0-SNAPSHOT (r1309051)
> > compiled Apr 03 2012, 11:18:53
> >
> > Here's the gist with stack traces, both with or without specifying
> schema.
> > Am using piggybank from trunk.
> >
> > https://gist.github.com/2293909
> >
> > Regards,
> >
> > Dan
> >
> >
> > On Tue, Apr 3, 2012 at 11:07 AM, Russell Jurney <
> russell.jurney@gmail.com
> > >wrote:
> >
> > > This looks like a bug fixed in 0.10. Mind trying it?
> > >
> > > Russell Jurney http://datasyndrome.com
> > >
> > > On Apr 3, 2012, at 9:13 AM, Dan Young <da...@gmail.com> wrote:
> > >
> > > > Hello Stan,
> > > >
> > > > I'm back from Mexico now, and here's my GIST with all the
> information.
> > > >
> > > > https://gist.github.com/2293226
> > > >
> > > > Any insight into what I'm not doing correctly would be greatly
> > > appreciated.
> > > >
> > > > Regards,
> > > >
> > > > Dan
> > > >
> > > > On Mon, Mar 26, 2012 at 9:11 AM, Stan Rosenberg <
> > > stan.rosenberg@gmail.com>wrote:
> > > >
> > > >> Hi Dan,
> > > >>
> > > >> Could you attach your script and sample input files for both cases
> > > >> (with and without the schema). In the case where no schema is
> > > >> provided, the stack trace shows that
> > > >> the 'pigSchema' which is passed into AvroStorageUtils.isTupleWrapper
> > > >> has no name, hence the NPE. I am not sure why that happens.
> > > >> I will investigate further once I can execute your scripts.
> > > >>
> > > >> Best,
> > > >>
> > > >> stan
> > > >>
> > > >> On Sun, Mar 25, 2012 at 10:41 AM, Stan Rosenberg
> > > >> <st...@gmail.com> wrote:
> > > >>> Hi Dan,
> > > >>>
> > > >>> This looks like an avro bug. I'll have a look later tonight unless
> > > >> someone
> > > >>> else has a more immediate answer.
> > > >>>
> > > >>> Best,
> > > >>>
> > > >>> stan
> > > >>>
> > > >>> On Mar 25, 2012 12:36 AM, "Dan Young" <da...@gmail.com> wrote:
> > > >>>>
> > > >>>> Hello all,
> > > >>>>
> > > >>>> I'm trying to store a bag of tuples using AvroStorage but am not
> > able
> > > to
> > > >>>> figure out what I'm doing wrong (or if it' supported). What I
> have
> > is
> > > >> the
> > > >>>> following:
> > > >>>>
> > > >>>> grunt>illustrate c;
> > > >>>> ....
> > > >>>> ....
> > > >>>>
> > > >>>>
> > > >>
> > >
> >
> ---------------------------------------------------------------------------------------------
> > > >>>> | c | franchise_id:int |
> > > >>>> cast_and_crew:bag{:tuple(full_name:chararray)}|
> > > >>>>
> > > >>>>
> > > >>
> > >
> >
> ---------------------------------------------------------------------------------------------
> > > >>>> | | 213939 | {(Wang Junzheng), (Li Ling)}
> > > >>>> |
> > > >>>>
> > > >>>>
> > > >>
> > >
> >
> ---------------------------------------------------------------------------------------------
> > > >>>>
> > > >>>> When I try to store is via AvroStorage I'm getting an error. I've
> > > tried
> > > >>>> both with and without specifying a schema I've created a gist and
> > > >> posted
> > > >>>> it here with the Pig Stack Trace.....
> > > >>>>
> > > >>>> https://gist.github.com/2191363
> > > >>>>
> > > >>>>
> > > >>>> If anyone has insight into what I'm doing wrong, and can help, I
> > would
> > > >>>> greatly appreciate the guidance.
> > > >>>>
> > > >>>>
> > > >>>> Regards,
> > > >>>>
> > > >>>> Dan
> > > >>
> > >
> >
>
>
>
> --
> *Note that I'm no longer using my Yahoo! email address. Please email me at
> billgraham@gmail.com going forward.*
>
Re: Trying to store a bag of tuples using AvroStorage.
Posted by Bill Graham <bi...@gmail.com>.
In the schema approach the error is that your json is invalid. You're
missing a second '}' before the last ']'.
On Tue, Apr 3, 2012 at 10:32 AM, Dan Young <da...@gmail.com> wrote:
> I just updated my pig from svn repo and now am using the latest from trunk:
>
> pig -i
> Apache Pig version 0.11.0-SNAPSHOT (r1309051)
> compiled Apr 03 2012, 11:18:53
>
> Here's the gist with stack traces, both with or without specifying schema.
> Am using piggybank from trunk.
>
> https://gist.github.com/2293909
>
> Regards,
>
> Dan
>
>
> On Tue, Apr 3, 2012 at 11:07 AM, Russell Jurney <russell.jurney@gmail.com
> >wrote:
>
> > This looks like a bug fixed in 0.10. Mind trying it?
> >
> > Russell Jurney http://datasyndrome.com
> >
> > On Apr 3, 2012, at 9:13 AM, Dan Young <da...@gmail.com> wrote:
> >
> > > Hello Stan,
> > >
> > > I'm back from Mexico now, and here's my GIST with all the information.
> > >
> > > https://gist.github.com/2293226
> > >
> > > Any insight into what I'm not doing correctly would be greatly
> > appreciated.
> > >
> > > Regards,
> > >
> > > Dan
> > >
> > > On Mon, Mar 26, 2012 at 9:11 AM, Stan Rosenberg <
> > stan.rosenberg@gmail.com>wrote:
> > >
> > >> Hi Dan,
> > >>
> > >> Could you attach your script and sample input files for both cases
> > >> (with and without the schema). In the case where no schema is
> > >> provided, the stack trace shows that
> > >> the 'pigSchema' which is passed into AvroStorageUtils.isTupleWrapper
> > >> has no name, hence the NPE. I am not sure why that happens.
> > >> I will investigate further once I can execute your scripts.
> > >>
> > >> Best,
> > >>
> > >> stan
> > >>
> > >> On Sun, Mar 25, 2012 at 10:41 AM, Stan Rosenberg
> > >> <st...@gmail.com> wrote:
> > >>> Hi Dan,
> > >>>
> > >>> This looks like an avro bug. I'll have a look later tonight unless
> > >> someone
> > >>> else has a more immediate answer.
> > >>>
> > >>> Best,
> > >>>
> > >>> stan
> > >>>
> > >>> On Mar 25, 2012 12:36 AM, "Dan Young" <da...@gmail.com> wrote:
> > >>>>
> > >>>> Hello all,
> > >>>>
> > >>>> I'm trying to store a bag of tuples using AvroStorage but am not
> able
> > to
> > >>>> figure out what I'm doing wrong (or if it' supported). What I have
> is
> > >> the
> > >>>> following:
> > >>>>
> > >>>> grunt>illustrate c;
> > >>>> ....
> > >>>> ....
> > >>>>
> > >>>>
> > >>
> >
> ---------------------------------------------------------------------------------------------
> > >>>> | c | franchise_id:int |
> > >>>> cast_and_crew:bag{:tuple(full_name:chararray)}|
> > >>>>
> > >>>>
> > >>
> >
> ---------------------------------------------------------------------------------------------
> > >>>> | | 213939 | {(Wang Junzheng), (Li Ling)}
> > >>>> |
> > >>>>
> > >>>>
> > >>
> >
> ---------------------------------------------------------------------------------------------
> > >>>>
> > >>>> When I try to store is via AvroStorage I'm getting an error. I've
> > tried
> > >>>> both with and without specifying a schema I've created a gist and
> > >> posted
> > >>>> it here with the Pig Stack Trace.....
> > >>>>
> > >>>> https://gist.github.com/2191363
> > >>>>
> > >>>>
> > >>>> If anyone has insight into what I'm doing wrong, and can help, I
> would
> > >>>> greatly appreciate the guidance.
> > >>>>
> > >>>>
> > >>>> Regards,
> > >>>>
> > >>>> Dan
> > >>
> >
>
--
*Note that I'm no longer using my Yahoo! email address. Please email me at
billgraham@gmail.com going forward.*
Re: Trying to store a bag of tuples using AvroStorage.
Posted by Dan Young <da...@gmail.com>.
I just updated my pig from svn repo and now am using the latest from trunk:
pig -i
Apache Pig version 0.11.0-SNAPSHOT (r1309051)
compiled Apr 03 2012, 11:18:53
Here's the gist with stack traces, both with or without specifying schema.
Am using piggybank from trunk.
https://gist.github.com/2293909
Regards,
Dan
On Tue, Apr 3, 2012 at 11:07 AM, Russell Jurney <ru...@gmail.com>wrote:
> This looks like a bug fixed in 0.10. Mind trying it?
>
> Russell Jurney http://datasyndrome.com
>
> On Apr 3, 2012, at 9:13 AM, Dan Young <da...@gmail.com> wrote:
>
> > Hello Stan,
> >
> > I'm back from Mexico now, and here's my GIST with all the information.
> >
> > https://gist.github.com/2293226
> >
> > Any insight into what I'm not doing correctly would be greatly
> appreciated.
> >
> > Regards,
> >
> > Dan
> >
> > On Mon, Mar 26, 2012 at 9:11 AM, Stan Rosenberg <
> stan.rosenberg@gmail.com>wrote:
> >
> >> Hi Dan,
> >>
> >> Could you attach your script and sample input files for both cases
> >> (with and without the schema). In the case where no schema is
> >> provided, the stack trace shows that
> >> the 'pigSchema' which is passed into AvroStorageUtils.isTupleWrapper
> >> has no name, hence the NPE. I am not sure why that happens.
> >> I will investigate further once I can execute your scripts.
> >>
> >> Best,
> >>
> >> stan
> >>
> >> On Sun, Mar 25, 2012 at 10:41 AM, Stan Rosenberg
> >> <st...@gmail.com> wrote:
> >>> Hi Dan,
> >>>
> >>> This looks like an avro bug. I'll have a look later tonight unless
> >> someone
> >>> else has a more immediate answer.
> >>>
> >>> Best,
> >>>
> >>> stan
> >>>
> >>> On Mar 25, 2012 12:36 AM, "Dan Young" <da...@gmail.com> wrote:
> >>>>
> >>>> Hello all,
> >>>>
> >>>> I'm trying to store a bag of tuples using AvroStorage but am not able
> to
> >>>> figure out what I'm doing wrong (or if it' supported). What I have is
> >> the
> >>>> following:
> >>>>
> >>>> grunt>illustrate c;
> >>>> ....
> >>>> ....
> >>>>
> >>>>
> >>
> ---------------------------------------------------------------------------------------------
> >>>> | c | franchise_id:int |
> >>>> cast_and_crew:bag{:tuple(full_name:chararray)}|
> >>>>
> >>>>
> >>
> ---------------------------------------------------------------------------------------------
> >>>> | | 213939 | {(Wang Junzheng), (Li Ling)}
> >>>> |
> >>>>
> >>>>
> >>
> ---------------------------------------------------------------------------------------------
> >>>>
> >>>> When I try to store is via AvroStorage I'm getting an error. I've
> tried
> >>>> both with and without specifying a schema I've created a gist and
> >> posted
> >>>> it here with the Pig Stack Trace.....
> >>>>
> >>>> https://gist.github.com/2191363
> >>>>
> >>>>
> >>>> If anyone has insight into what I'm doing wrong, and can help, I would
> >>>> greatly appreciate the guidance.
> >>>>
> >>>>
> >>>> Regards,
> >>>>
> >>>> Dan
> >>
>
Re: Trying to store a bag of tuples using AvroStorage.
Posted by Dan Young <da...@gmail.com>.
Here's the version of Pig I'm using:
pig -i
Apache Pig version 0.11.0-SNAPSHOT (r1304979)
compiled Mar 24 2012, 21:48:44
The version of Hadoop:
*Version:* 1.0.0, r1214675
Regards,
Dan
On Tue, Apr 3, 2012 at 11:07 AM, Russell Jurney <ru...@gmail.com>wrote:
> This looks like a bug fixed in 0.10. Mind trying it?
>
> Russell Jurney http://datasyndrome.com
>
> On Apr 3, 2012, at 9:13 AM, Dan Young <da...@gmail.com> wrote:
>
> > Hello Stan,
> >
> > I'm back from Mexico now, and here's my GIST with all the information.
> >
> > https://gist.github.com/2293226
> >
> > Any insight into what I'm not doing correctly would be greatly
> appreciated.
> >
> > Regards,
> >
> > Dan
> >
> > On Mon, Mar 26, 2012 at 9:11 AM, Stan Rosenberg <
> stan.rosenberg@gmail.com>wrote:
> >
> >> Hi Dan,
> >>
> >> Could you attach your script and sample input files for both cases
> >> (with and without the schema). In the case where no schema is
> >> provided, the stack trace shows that
> >> the 'pigSchema' which is passed into AvroStorageUtils.isTupleWrapper
> >> has no name, hence the NPE. I am not sure why that happens.
> >> I will investigate further once I can execute your scripts.
> >>
> >> Best,
> >>
> >> stan
> >>
> >> On Sun, Mar 25, 2012 at 10:41 AM, Stan Rosenberg
> >> <st...@gmail.com> wrote:
> >>> Hi Dan,
> >>>
> >>> This looks like an avro bug. I'll have a look later tonight unless
> >> someone
> >>> else has a more immediate answer.
> >>>
> >>> Best,
> >>>
> >>> stan
> >>>
> >>> On Mar 25, 2012 12:36 AM, "Dan Young" <da...@gmail.com> wrote:
> >>>>
> >>>> Hello all,
> >>>>
> >>>> I'm trying to store a bag of tuples using AvroStorage but am not able
> to
> >>>> figure out what I'm doing wrong (or if it' supported). What I have is
> >> the
> >>>> following:
> >>>>
> >>>> grunt>illustrate c;
> >>>> ....
> >>>> ....
> >>>>
> >>>>
> >>
> ---------------------------------------------------------------------------------------------
> >>>> | c | franchise_id:int |
> >>>> cast_and_crew:bag{:tuple(full_name:chararray)}|
> >>>>
> >>>>
> >>
> ---------------------------------------------------------------------------------------------
> >>>> | | 213939 | {(Wang Junzheng), (Li Ling)}
> >>>> |
> >>>>
> >>>>
> >>
> ---------------------------------------------------------------------------------------------
> >>>>
> >>>> When I try to store is via AvroStorage I'm getting an error. I've
> tried
> >>>> both with and without specifying a schema I've created a gist and
> >> posted
> >>>> it here with the Pig Stack Trace.....
> >>>>
> >>>> https://gist.github.com/2191363
> >>>>
> >>>>
> >>>> If anyone has insight into what I'm doing wrong, and can help, I would
> >>>> greatly appreciate the guidance.
> >>>>
> >>>>
> >>>> Regards,
> >>>>
> >>>> Dan
> >>
>
Re: Trying to store a bag of tuples using AvroStorage.
Posted by Russell Jurney <ru...@gmail.com>.
This looks like a bug fixed in 0.10. Mind trying it?
Russell Jurney http://datasyndrome.com
On Apr 3, 2012, at 9:13 AM, Dan Young <da...@gmail.com> wrote:
> Hello Stan,
>
> I'm back from Mexico now, and here's my GIST with all the information.
>
> https://gist.github.com/2293226
>
> Any insight into what I'm not doing correctly would be greatly appreciated.
>
> Regards,
>
> Dan
>
> On Mon, Mar 26, 2012 at 9:11 AM, Stan Rosenberg <st...@gmail.com>wrote:
>
>> Hi Dan,
>>
>> Could you attach your script and sample input files for both cases
>> (with and without the schema). In the case where no schema is
>> provided, the stack trace shows that
>> the 'pigSchema' which is passed into AvroStorageUtils.isTupleWrapper
>> has no name, hence the NPE. I am not sure why that happens.
>> I will investigate further once I can execute your scripts.
>>
>> Best,
>>
>> stan
>>
>> On Sun, Mar 25, 2012 at 10:41 AM, Stan Rosenberg
>> <st...@gmail.com> wrote:
>>> Hi Dan,
>>>
>>> This looks like an avro bug. I'll have a look later tonight unless
>> someone
>>> else has a more immediate answer.
>>>
>>> Best,
>>>
>>> stan
>>>
>>> On Mar 25, 2012 12:36 AM, "Dan Young" <da...@gmail.com> wrote:
>>>>
>>>> Hello all,
>>>>
>>>> I'm trying to store a bag of tuples using AvroStorage but am not able to
>>>> figure out what I'm doing wrong (or if it' supported). What I have is
>> the
>>>> following:
>>>>
>>>> grunt>illustrate c;
>>>> ....
>>>> ....
>>>>
>>>>
>> ---------------------------------------------------------------------------------------------
>>>> | c | franchise_id:int |
>>>> cast_and_crew:bag{:tuple(full_name:chararray)}|
>>>>
>>>>
>> ---------------------------------------------------------------------------------------------
>>>> | | 213939 | {(Wang Junzheng), (Li Ling)}
>>>> |
>>>>
>>>>
>> ---------------------------------------------------------------------------------------------
>>>>
>>>> When I try to store is via AvroStorage I'm getting an error. I've tried
>>>> both with and without specifying a schema I've created a gist and
>> posted
>>>> it here with the Pig Stack Trace.....
>>>>
>>>> https://gist.github.com/2191363
>>>>
>>>>
>>>> If anyone has insight into what I'm doing wrong, and can help, I would
>>>> greatly appreciate the guidance.
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Dan
>>
Re: Trying to store a bag of tuples using AvroStorage.
Posted by Dan Young <da...@gmail.com>.
Hello Stan,
I'm back from Mexico now, and here's my GIST with all the information.
https://gist.github.com/2293226
Any insight into what I'm not doing correctly would be greatly appreciated.
Regards,
Dan
On Mon, Mar 26, 2012 at 9:11 AM, Stan Rosenberg <st...@gmail.com>wrote:
> Hi Dan,
>
> Could you attach your script and sample input files for both cases
> (with and without the schema). In the case where no schema is
> provided, the stack trace shows that
> the 'pigSchema' which is passed into AvroStorageUtils.isTupleWrapper
> has no name, hence the NPE. I am not sure why that happens.
> I will investigate further once I can execute your scripts.
>
> Best,
>
> stan
>
> On Sun, Mar 25, 2012 at 10:41 AM, Stan Rosenberg
> <st...@gmail.com> wrote:
> > Hi Dan,
> >
> > This looks like an avro bug. I'll have a look later tonight unless
> someone
> > else has a more immediate answer.
> >
> > Best,
> >
> > stan
> >
> > On Mar 25, 2012 12:36 AM, "Dan Young" <da...@gmail.com> wrote:
> >>
> >> Hello all,
> >>
> >> I'm trying to store a bag of tuples using AvroStorage but am not able to
> >> figure out what I'm doing wrong (or if it' supported). What I have is
> the
> >> following:
> >>
> >> grunt>illustrate c;
> >> ....
> >> ....
> >>
> >>
> ---------------------------------------------------------------------------------------------
> >> | c | franchise_id:int |
> >> cast_and_crew:bag{:tuple(full_name:chararray)}|
> >>
> >>
> ---------------------------------------------------------------------------------------------
> >> | | 213939 | {(Wang Junzheng), (Li Ling)}
> >> |
> >>
> >>
> ---------------------------------------------------------------------------------------------
> >>
> >> When I try to store is via AvroStorage I'm getting an error. I've tried
> >> both with and without specifying a schema I've created a gist and
> posted
> >> it here with the Pig Stack Trace.....
> >>
> >> https://gist.github.com/2191363
> >>
> >>
> >> If anyone has insight into what I'm doing wrong, and can help, I would
> >> greatly appreciate the guidance.
> >>
> >>
> >> Regards,
> >>
> >> Dan
>
Re: Trying to store a bag of tuples using AvroStorage.
Posted by Stan Rosenberg <st...@gmail.com>.
Hi Dan,
Could you attach your script and sample input files for both cases
(with and without the schema). In the case where no schema is
provided, the stack trace shows that
the 'pigSchema' which is passed into AvroStorageUtils.isTupleWrapper
has no name, hence the NPE. I am not sure why that happens.
I will investigate further once I can execute your scripts.
Best,
stan
On Sun, Mar 25, 2012 at 10:41 AM, Stan Rosenberg
<st...@gmail.com> wrote:
> Hi Dan,
>
> This looks like an avro bug. I'll have a look later tonight unless someone
> else has a more immediate answer.
>
> Best,
>
> stan
>
> On Mar 25, 2012 12:36 AM, "Dan Young" <da...@gmail.com> wrote:
>>
>> Hello all,
>>
>> I'm trying to store a bag of tuples using AvroStorage but am not able to
>> figure out what I'm doing wrong (or if it' supported). What I have is the
>> following:
>>
>> grunt>illustrate c;
>> ....
>> ....
>>
>> ---------------------------------------------------------------------------------------------
>> | c | franchise_id:int |
>> cast_and_crew:bag{:tuple(full_name:chararray)}|
>>
>> ---------------------------------------------------------------------------------------------
>> | | 213939 | {(Wang Junzheng), (Li Ling)}
>> |
>>
>> ---------------------------------------------------------------------------------------------
>>
>> When I try to store is via AvroStorage I'm getting an error. I've tried
>> both with and without specifying a schema I've created a gist and posted
>> it here with the Pig Stack Trace.....
>>
>> https://gist.github.com/2191363
>>
>>
>> If anyone has insight into what I'm doing wrong, and can help, I would
>> greatly appreciate the guidance.
>>
>>
>> Regards,
>>
>> Dan
Re: Trying to store a bag of tuples using AvroStorage.
Posted by Stan Rosenberg <st...@gmail.com>.
Hi Dan,
This looks like an avro bug. I'll have a look later tonight unless someone
else has a more immediate answer.
Best,
stan
On Mar 25, 2012 12:36 AM, "Dan Young" <da...@gmail.com> wrote:
> Hello all,
>
> I'm trying to store a bag of tuples using AvroStorage but am not able to
> figure out what I'm doing wrong (or if it' supported). What I have is the
> following:
>
> grunt>illustrate c;
> ....
> ....
>
> ---------------------------------------------------------------------------------------------
> | c | franchise_id:int |
> cast_and_crew:bag{:tuple(full_name:chararray)}|
>
> ---------------------------------------------------------------------------------------------
> | | 213939 | {(Wang Junzheng), (Li Ling)}
> |
>
> ---------------------------------------------------------------------------------------------
>
> When I try to store is via AvroStorage I'm getting an error. I've tried
> both with and without specifying a schema I've created a gist and posted
> it here with the Pig Stack Trace.....
>
> https://gist.github.com/2191363
>
>
> If anyone has insight into what I'm doing wrong, and can help, I would
> greatly appreciate the guidance.
>
>
> Regards,
>
> Dan
>