You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Dan Young <da...@gmail.com> on 2012/04/03 18:13:20 UTC
Re: Trying to store a bag of tuples using AvroStorage.
Hello Stan,
I'm back from Mexico now, and here's my GIST with all the information.
https://gist.github.com/2293226
Any insight into what I'm not doing correctly would be greatly appreciated.
Regards,
Dan
On Mon, Mar 26, 2012 at 9:11 AM, Stan Rosenberg <st...@gmail.com>wrote:
> Hi Dan,
>
> Could you attach your script and sample input files for both cases
> (with and without the schema). In the case where no schema is
> provided, the stack trace shows that
> the 'pigSchema' which is passed into AvroStorageUtils.isTupleWrapper
> has no name, hence the NPE. I am not sure why that happens.
> I will investigate further once I can execute your scripts.
>
> Best,
>
> stan
>
> On Sun, Mar 25, 2012 at 10:41 AM, Stan Rosenberg
> <st...@gmail.com> wrote:
> > Hi Dan,
> >
> > This looks like an avro bug. I'll have a look later tonight unless
> someone
> > else has a more immediate answer.
> >
> > Best,
> >
> > stan
> >
> > On Mar 25, 2012 12:36 AM, "Dan Young" <da...@gmail.com> wrote:
> >>
> >> Hello all,
> >>
> >> I'm trying to store a bag of tuples using AvroStorage but am not able to
> >> figure out what I'm doing wrong (or if it' supported). What I have is
> the
> >> following:
> >>
> >> grunt>illustrate c;
> >> ....
> >> ....
> >>
> >>
> ---------------------------------------------------------------------------------------------
> >> | c | franchise_id:int |
> >> cast_and_crew:bag{:tuple(full_name:chararray)}|
> >>
> >>
> ---------------------------------------------------------------------------------------------
> >> | | 213939 | {(Wang Junzheng), (Li Ling)}
> >> |
> >>
> >>
> ---------------------------------------------------------------------------------------------
> >>
> >> When I try to store is via AvroStorage I'm getting an error. I've tried
> >> both with and without specifying a schema I've created a gist and
> posted
> >> it here with the Pig Stack Trace.....
> >>
> >> https://gist.github.com/2191363
> >>
> >>
> >> If anyone has insight into what I'm doing wrong, and can help, I would
> >> greatly appreciate the guidance.
> >>
> >>
> >> Regards,
> >>
> >> Dan
>
Re: Trying to store a bag of tuples using AvroStorage.
Posted by Dan Young <da...@gmail.com>.
Dooohh....thank you for pointing that out....I thought I ran that thru
jsonlint.....That seemed to fix it........
Regards,
Dano
On Tue, Apr 3, 2012 at 12:11 PM, Bill Graham <bi...@gmail.com> wrote:
> In the schema approach the error is that your json is invalid. You're
> missing a second '}' before the last ']'.
>
>
>
> On Tue, Apr 3, 2012 at 10:32 AM, Dan Young <da...@gmail.com> wrote:
>
> > I just updated my pig from svn repo and now am using the latest from
> trunk:
> >
> > pig -i
> > Apache Pig version 0.11.0-SNAPSHOT (r1309051)
> > compiled Apr 03 2012, 11:18:53
> >
> > Here's the gist with stack traces, both with or without specifying
> schema.
> > Am using piggybank from trunk.
> >
> > https://gist.github.com/2293909
> >
> > Regards,
> >
> > Dan
> >
> >
> > On Tue, Apr 3, 2012 at 11:07 AM, Russell Jurney <
> russell.jurney@gmail.com
> > >wrote:
> >
> > > This looks like a bug fixed in 0.10. Mind trying it?
> > >
> > > Russell Jurney http://datasyndrome.com
> > >
> > > On Apr 3, 2012, at 9:13 AM, Dan Young <da...@gmail.com> wrote:
> > >
> > > > Hello Stan,
> > > >
> > > > I'm back from Mexico now, and here's my GIST with all the
> information.
> > > >
> > > > https://gist.github.com/2293226
> > > >
> > > > Any insight into what I'm not doing correctly would be greatly
> > > appreciated.
> > > >
> > > > Regards,
> > > >
> > > > Dan
> > > >
> > > > On Mon, Mar 26, 2012 at 9:11 AM, Stan Rosenberg <
> > > stan.rosenberg@gmail.com>wrote:
> > > >
> > > >> Hi Dan,
> > > >>
> > > >> Could you attach your script and sample input files for both cases
> > > >> (with and without the schema). In the case where no schema is
> > > >> provided, the stack trace shows that
> > > >> the 'pigSchema' which is passed into AvroStorageUtils.isTupleWrapper
> > > >> has no name, hence the NPE. I am not sure why that happens.
> > > >> I will investigate further once I can execute your scripts.
> > > >>
> > > >> Best,
> > > >>
> > > >> stan
> > > >>
> > > >> On Sun, Mar 25, 2012 at 10:41 AM, Stan Rosenberg
> > > >> <st...@gmail.com> wrote:
> > > >>> Hi Dan,
> > > >>>
> > > >>> This looks like an avro bug. I'll have a look later tonight unless
> > > >> someone
> > > >>> else has a more immediate answer.
> > > >>>
> > > >>> Best,
> > > >>>
> > > >>> stan
> > > >>>
> > > >>> On Mar 25, 2012 12:36 AM, "Dan Young" <da...@gmail.com> wrote:
> > > >>>>
> > > >>>> Hello all,
> > > >>>>
> > > >>>> I'm trying to store a bag of tuples using AvroStorage but am not
> > able
> > > to
> > > >>>> figure out what I'm doing wrong (or if it' supported). What I
> have
> > is
> > > >> the
> > > >>>> following:
> > > >>>>
> > > >>>> grunt>illustrate c;
> > > >>>> ....
> > > >>>> ....
> > > >>>>
> > > >>>>
> > > >>
> > >
> >
> ---------------------------------------------------------------------------------------------
> > > >>>> | c | franchise_id:int |
> > > >>>> cast_and_crew:bag{:tuple(full_name:chararray)}|
> > > >>>>
> > > >>>>
> > > >>
> > >
> >
> ---------------------------------------------------------------------------------------------
> > > >>>> | | 213939 | {(Wang Junzheng), (Li Ling)}
> > > >>>> |
> > > >>>>
> > > >>>>
> > > >>
> > >
> >
> ---------------------------------------------------------------------------------------------
> > > >>>>
> > > >>>> When I try to store is via AvroStorage I'm getting an error. I've
> > > tried
> > > >>>> both with and without specifying a schema I've created a gist and
> > > >> posted
> > > >>>> it here with the Pig Stack Trace.....
> > > >>>>
> > > >>>> https://gist.github.com/2191363
> > > >>>>
> > > >>>>
> > > >>>> If anyone has insight into what I'm doing wrong, and can help, I
> > would
> > > >>>> greatly appreciate the guidance.
> > > >>>>
> > > >>>>
> > > >>>> Regards,
> > > >>>>
> > > >>>> Dan
> > > >>
> > >
> >
>
>
>
> --
> *Note that I'm no longer using my Yahoo! email address. Please email me at
> billgraham@gmail.com going forward.*
>
Re: Trying to store a bag of tuples using AvroStorage.
Posted by Bill Graham <bi...@gmail.com>.
In the schema approach the error is that your json is invalid. You're
missing a second '}' before the last ']'.
On Tue, Apr 3, 2012 at 10:32 AM, Dan Young <da...@gmail.com> wrote:
> I just updated my pig from svn repo and now am using the latest from trunk:
>
> pig -i
> Apache Pig version 0.11.0-SNAPSHOT (r1309051)
> compiled Apr 03 2012, 11:18:53
>
> Here's the gist with stack traces, both with or without specifying schema.
> Am using piggybank from trunk.
>
> https://gist.github.com/2293909
>
> Regards,
>
> Dan
>
>
> On Tue, Apr 3, 2012 at 11:07 AM, Russell Jurney <russell.jurney@gmail.com
> >wrote:
>
> > This looks like a bug fixed in 0.10. Mind trying it?
> >
> > Russell Jurney http://datasyndrome.com
> >
> > On Apr 3, 2012, at 9:13 AM, Dan Young <da...@gmail.com> wrote:
> >
> > > Hello Stan,
> > >
> > > I'm back from Mexico now, and here's my GIST with all the information.
> > >
> > > https://gist.github.com/2293226
> > >
> > > Any insight into what I'm not doing correctly would be greatly
> > appreciated.
> > >
> > > Regards,
> > >
> > > Dan
> > >
> > > On Mon, Mar 26, 2012 at 9:11 AM, Stan Rosenberg <
> > stan.rosenberg@gmail.com>wrote:
> > >
> > >> Hi Dan,
> > >>
> > >> Could you attach your script and sample input files for both cases
> > >> (with and without the schema). In the case where no schema is
> > >> provided, the stack trace shows that
> > >> the 'pigSchema' which is passed into AvroStorageUtils.isTupleWrapper
> > >> has no name, hence the NPE. I am not sure why that happens.
> > >> I will investigate further once I can execute your scripts.
> > >>
> > >> Best,
> > >>
> > >> stan
> > >>
> > >> On Sun, Mar 25, 2012 at 10:41 AM, Stan Rosenberg
> > >> <st...@gmail.com> wrote:
> > >>> Hi Dan,
> > >>>
> > >>> This looks like an avro bug. I'll have a look later tonight unless
> > >> someone
> > >>> else has a more immediate answer.
> > >>>
> > >>> Best,
> > >>>
> > >>> stan
> > >>>
> > >>> On Mar 25, 2012 12:36 AM, "Dan Young" <da...@gmail.com> wrote:
> > >>>>
> > >>>> Hello all,
> > >>>>
> > >>>> I'm trying to store a bag of tuples using AvroStorage but am not
> able
> > to
> > >>>> figure out what I'm doing wrong (or if it' supported). What I have
> is
> > >> the
> > >>>> following:
> > >>>>
> > >>>> grunt>illustrate c;
> > >>>> ....
> > >>>> ....
> > >>>>
> > >>>>
> > >>
> >
> ---------------------------------------------------------------------------------------------
> > >>>> | c | franchise_id:int |
> > >>>> cast_and_crew:bag{:tuple(full_name:chararray)}|
> > >>>>
> > >>>>
> > >>
> >
> ---------------------------------------------------------------------------------------------
> > >>>> | | 213939 | {(Wang Junzheng), (Li Ling)}
> > >>>> |
> > >>>>
> > >>>>
> > >>
> >
> ---------------------------------------------------------------------------------------------
> > >>>>
> > >>>> When I try to store is via AvroStorage I'm getting an error. I've
> > tried
> > >>>> both with and without specifying a schema I've created a gist and
> > >> posted
> > >>>> it here with the Pig Stack Trace.....
> > >>>>
> > >>>> https://gist.github.com/2191363
> > >>>>
> > >>>>
> > >>>> If anyone has insight into what I'm doing wrong, and can help, I
> would
> > >>>> greatly appreciate the guidance.
> > >>>>
> > >>>>
> > >>>> Regards,
> > >>>>
> > >>>> Dan
> > >>
> >
>
--
*Note that I'm no longer using my Yahoo! email address. Please email me at
billgraham@gmail.com going forward.*
Re: Trying to store a bag of tuples using AvroStorage.
Posted by Dan Young <da...@gmail.com>.
I just updated my pig from svn repo and now am using the latest from trunk:
pig -i
Apache Pig version 0.11.0-SNAPSHOT (r1309051)
compiled Apr 03 2012, 11:18:53
Here's the gist with stack traces, both with or without specifying schema.
Am using piggybank from trunk.
https://gist.github.com/2293909
Regards,
Dan
On Tue, Apr 3, 2012 at 11:07 AM, Russell Jurney <ru...@gmail.com>wrote:
> This looks like a bug fixed in 0.10. Mind trying it?
>
> Russell Jurney http://datasyndrome.com
>
> On Apr 3, 2012, at 9:13 AM, Dan Young <da...@gmail.com> wrote:
>
> > Hello Stan,
> >
> > I'm back from Mexico now, and here's my GIST with all the information.
> >
> > https://gist.github.com/2293226
> >
> > Any insight into what I'm not doing correctly would be greatly
> appreciated.
> >
> > Regards,
> >
> > Dan
> >
> > On Mon, Mar 26, 2012 at 9:11 AM, Stan Rosenberg <
> stan.rosenberg@gmail.com>wrote:
> >
> >> Hi Dan,
> >>
> >> Could you attach your script and sample input files for both cases
> >> (with and without the schema). In the case where no schema is
> >> provided, the stack trace shows that
> >> the 'pigSchema' which is passed into AvroStorageUtils.isTupleWrapper
> >> has no name, hence the NPE. I am not sure why that happens.
> >> I will investigate further once I can execute your scripts.
> >>
> >> Best,
> >>
> >> stan
> >>
> >> On Sun, Mar 25, 2012 at 10:41 AM, Stan Rosenberg
> >> <st...@gmail.com> wrote:
> >>> Hi Dan,
> >>>
> >>> This looks like an avro bug. I'll have a look later tonight unless
> >> someone
> >>> else has a more immediate answer.
> >>>
> >>> Best,
> >>>
> >>> stan
> >>>
> >>> On Mar 25, 2012 12:36 AM, "Dan Young" <da...@gmail.com> wrote:
> >>>>
> >>>> Hello all,
> >>>>
> >>>> I'm trying to store a bag of tuples using AvroStorage but am not able
> to
> >>>> figure out what I'm doing wrong (or if it' supported). What I have is
> >> the
> >>>> following:
> >>>>
> >>>> grunt>illustrate c;
> >>>> ....
> >>>> ....
> >>>>
> >>>>
> >>
> ---------------------------------------------------------------------------------------------
> >>>> | c | franchise_id:int |
> >>>> cast_and_crew:bag{:tuple(full_name:chararray)}|
> >>>>
> >>>>
> >>
> ---------------------------------------------------------------------------------------------
> >>>> | | 213939 | {(Wang Junzheng), (Li Ling)}
> >>>> |
> >>>>
> >>>>
> >>
> ---------------------------------------------------------------------------------------------
> >>>>
> >>>> When I try to store is via AvroStorage I'm getting an error. I've
> tried
> >>>> both with and without specifying a schema I've created a gist and
> >> posted
> >>>> it here with the Pig Stack Trace.....
> >>>>
> >>>> https://gist.github.com/2191363
> >>>>
> >>>>
> >>>> If anyone has insight into what I'm doing wrong, and can help, I would
> >>>> greatly appreciate the guidance.
> >>>>
> >>>>
> >>>> Regards,
> >>>>
> >>>> Dan
> >>
>
Re: Trying to store a bag of tuples using AvroStorage.
Posted by Dan Young <da...@gmail.com>.
Here's the version of Pig I'm using:
pig -i
Apache Pig version 0.11.0-SNAPSHOT (r1304979)
compiled Mar 24 2012, 21:48:44
The version of Hadoop:
*Version:* 1.0.0, r1214675
Regards,
Dan
On Tue, Apr 3, 2012 at 11:07 AM, Russell Jurney <ru...@gmail.com>wrote:
> This looks like a bug fixed in 0.10. Mind trying it?
>
> Russell Jurney http://datasyndrome.com
>
> On Apr 3, 2012, at 9:13 AM, Dan Young <da...@gmail.com> wrote:
>
> > Hello Stan,
> >
> > I'm back from Mexico now, and here's my GIST with all the information.
> >
> > https://gist.github.com/2293226
> >
> > Any insight into what I'm not doing correctly would be greatly
> appreciated.
> >
> > Regards,
> >
> > Dan
> >
> > On Mon, Mar 26, 2012 at 9:11 AM, Stan Rosenberg <
> stan.rosenberg@gmail.com>wrote:
> >
> >> Hi Dan,
> >>
> >> Could you attach your script and sample input files for both cases
> >> (with and without the schema). In the case where no schema is
> >> provided, the stack trace shows that
> >> the 'pigSchema' which is passed into AvroStorageUtils.isTupleWrapper
> >> has no name, hence the NPE. I am not sure why that happens.
> >> I will investigate further once I can execute your scripts.
> >>
> >> Best,
> >>
> >> stan
> >>
> >> On Sun, Mar 25, 2012 at 10:41 AM, Stan Rosenberg
> >> <st...@gmail.com> wrote:
> >>> Hi Dan,
> >>>
> >>> This looks like an avro bug. I'll have a look later tonight unless
> >> someone
> >>> else has a more immediate answer.
> >>>
> >>> Best,
> >>>
> >>> stan
> >>>
> >>> On Mar 25, 2012 12:36 AM, "Dan Young" <da...@gmail.com> wrote:
> >>>>
> >>>> Hello all,
> >>>>
> >>>> I'm trying to store a bag of tuples using AvroStorage but am not able
> to
> >>>> figure out what I'm doing wrong (or if it' supported). What I have is
> >> the
> >>>> following:
> >>>>
> >>>> grunt>illustrate c;
> >>>> ....
> >>>> ....
> >>>>
> >>>>
> >>
> ---------------------------------------------------------------------------------------------
> >>>> | c | franchise_id:int |
> >>>> cast_and_crew:bag{:tuple(full_name:chararray)}|
> >>>>
> >>>>
> >>
> ---------------------------------------------------------------------------------------------
> >>>> | | 213939 | {(Wang Junzheng), (Li Ling)}
> >>>> |
> >>>>
> >>>>
> >>
> ---------------------------------------------------------------------------------------------
> >>>>
> >>>> When I try to store is via AvroStorage I'm getting an error. I've
> tried
> >>>> both with and without specifying a schema I've created a gist and
> >> posted
> >>>> it here with the Pig Stack Trace.....
> >>>>
> >>>> https://gist.github.com/2191363
> >>>>
> >>>>
> >>>> If anyone has insight into what I'm doing wrong, and can help, I would
> >>>> greatly appreciate the guidance.
> >>>>
> >>>>
> >>>> Regards,
> >>>>
> >>>> Dan
> >>
>
Re: Trying to store a bag of tuples using AvroStorage.
Posted by Russell Jurney <ru...@gmail.com>.
This looks like a bug fixed in 0.10. Mind trying it?
Russell Jurney http://datasyndrome.com
On Apr 3, 2012, at 9:13 AM, Dan Young <da...@gmail.com> wrote:
> Hello Stan,
>
> I'm back from Mexico now, and here's my GIST with all the information.
>
> https://gist.github.com/2293226
>
> Any insight into what I'm not doing correctly would be greatly appreciated.
>
> Regards,
>
> Dan
>
> On Mon, Mar 26, 2012 at 9:11 AM, Stan Rosenberg <st...@gmail.com>wrote:
>
>> Hi Dan,
>>
>> Could you attach your script and sample input files for both cases
>> (with and without the schema). In the case where no schema is
>> provided, the stack trace shows that
>> the 'pigSchema' which is passed into AvroStorageUtils.isTupleWrapper
>> has no name, hence the NPE. I am not sure why that happens.
>> I will investigate further once I can execute your scripts.
>>
>> Best,
>>
>> stan
>>
>> On Sun, Mar 25, 2012 at 10:41 AM, Stan Rosenberg
>> <st...@gmail.com> wrote:
>>> Hi Dan,
>>>
>>> This looks like an avro bug. I'll have a look later tonight unless
>> someone
>>> else has a more immediate answer.
>>>
>>> Best,
>>>
>>> stan
>>>
>>> On Mar 25, 2012 12:36 AM, "Dan Young" <da...@gmail.com> wrote:
>>>>
>>>> Hello all,
>>>>
>>>> I'm trying to store a bag of tuples using AvroStorage but am not able to
>>>> figure out what I'm doing wrong (or if it' supported). What I have is
>> the
>>>> following:
>>>>
>>>> grunt>illustrate c;
>>>> ....
>>>> ....
>>>>
>>>>
>> ---------------------------------------------------------------------------------------------
>>>> | c | franchise_id:int |
>>>> cast_and_crew:bag{:tuple(full_name:chararray)}|
>>>>
>>>>
>> ---------------------------------------------------------------------------------------------
>>>> | | 213939 | {(Wang Junzheng), (Li Ling)}
>>>> |
>>>>
>>>>
>> ---------------------------------------------------------------------------------------------
>>>>
>>>> When I try to store is via AvroStorage I'm getting an error. I've tried
>>>> both with and without specifying a schema I've created a gist and
>> posted
>>>> it here with the Pig Stack Trace.....
>>>>
>>>> https://gist.github.com/2191363
>>>>
>>>>
>>>> If anyone has insight into what I'm doing wrong, and can help, I would
>>>> greatly appreciate the guidance.
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Dan
>>