You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Nerius Landys <nl...@gmail.com> on 2012/05/22 01:40:06 UTC
Design issue, need feedback
I'm trying to do something with Pig that I believe Pig wasn't really
designed/intended to handle.
Normally, the way I'd do things with Pig is by feeding it data like so:
Fred 23
Adam 25
Mary 21
...where the columns are name and age.
Now, a requirement came up to have the Pig script be more flexible
when the format of the data changes. In fact with the system I'm
working in, the format of the data will change quite often. Let me
explain. So now in one scenario the data incoming would be this,
literally:
Name=Fred Age=23
Name=Adam Age=25
Name=Mary Age=21
...and if the format of the data changes somewhat it may appear like
this for example:
Age=23 Sex=male Name=Fred
Age=25 Sex=male Name=Adam
Age=21 Sex=female Name=Mary
<All data is tab delimited of course.>
However in all cases the only data I'm interested in is name and age,
in that order. And in all cases these two pieces of data are
guaranteed to appear.
So my question is, how easy and efficient would it be to write
something in the Pig langauge that transforms the key/value rows into
the format in my first example with just name and age columns?
That is, I want to transform, using a fixed series of Pig statements,
this type of row:
Name=Fred Age=23
to this:
Fred 23
And I want the same series of Pig statements to transform this type of row:
Age=23 Sex=male Name=Fred
to this:
Fred 23
How would I go about doing this, and would this be terribly
inefficient? Is this just not the way Pig was meant to work?
I can think of a way to do this using a UDF maybe but is there a way
to do this using builtins?
Thanks, Nerius
Re: Design issue, need feedback
Posted by Andy Schlaikjer <an...@gmail.com>.
Another possible solution: Use json for your storage and load via
JsonLoader:
https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/pig/load/JsonLoader.java
Then you could project the fields you'd like to use out of the loaded maps
via Pig's map dereference operator:
http://pig.apache.org/docs/r0.9.2/basic.html#deref
Andy
On Tue, May 22, 2012 at 4:43 AM, Ruslan Al-fakikh <
ruslan.al-fakikh@jalent.ru> wrote:
> Hey Nerius,
>
> As for the columns number changes - yes, Avro or Thrift can handle that.
> As for transforming a value of a row from this 'Age=23' to this '23' -
> this is what Pig can do for you.
> Try something like
> b = foreach a generate substring(0,4,$1) AS Age --I haven't tested it,
> there can be typos
> Or maybe some other builtins can do, like regex stuff
>
> Ruslan Al-Fakikh
>
> -----Original Message-----
> From: Jonathan Coveney [mailto:jcoveney@gmail.com]
> Sent: Tuesday, May 22, 2012 3:47 AM
> To: user@pig.apache.org
> Subject: Re: Design issue, need feedback
>
> There are a couple of ways that you can do this. One, is that you could
> make a special loader that converts your format to a map of (key,value)
> pairs, and then you can project however you want.
>
> Another (better, if at all possible) way would be to use something like
> Avro or Thrift that allows you to update your schemas in backwards
> compatible ways. I highly recommend leveraging one of these projects
> instead of rolling your own janky file format. Starting from nothing Avro
> would probably be easier, though here at Twitter we have standardized on
> Thrift. Pig can work with both.
>
> 2012/5/21 Nerius Landys <nl...@gmail.com>
>
> > I'm trying to do something with Pig that I believe Pig wasn't really
> > designed/intended to handle.
> > Normally, the way I'd do things with Pig is by feeding it data like so:
> >
> > Fred 23
> > Adam 25
> > Mary 21
> >
> > ...where the columns are name and age.
> >
> > Now, a requirement came up to have the Pig script be more flexible
> > when the format of the data changes. In fact with the system I'm
> > working in, the format of the data will change quite often. Let me
> > explain. So now in one scenario the data incoming would be this,
> > literally:
> >
> > Name=Fred Age=23
> > Name=Adam Age=25
> > Name=Mary Age=21
> >
> > ...and if the format of the data changes somewhat it may appear like
> > this for example:
> >
> > Age=23 Sex=male Name=Fred
> > Age=25 Sex=male Name=Adam
> > Age=21 Sex=female Name=Mary
> >
> > <All data is tab delimited of course.> However in all cases the only
> > data I'm interested in is name and age, in that order. And in all
> > cases these two pieces of data are guaranteed to appear.
> >
> > So my question is, how easy and efficient would it be to write
> > something in the Pig langauge that transforms the key/value rows into
> > the format in my first example with just name and age columns?
> >
> > That is, I want to transform, using a fixed series of Pig statements,
> > this type of row:
> >
> > Name=Fred Age=23
> >
> > to this:
> >
> > Fred 23
> >
> > And I want the same series of Pig statements to transform this type of
> row:
> >
> > Age=23 Sex=male Name=Fred
> >
> > to this:
> >
> > Fred 23
> >
> > How would I go about doing this, and would this be terribly
> > inefficient? Is this just not the way Pig was meant to work?
> > I can think of a way to do this using a UDF maybe but is there a way
> > to do this using builtins?
> >
> > Thanks, Nerius
> >
>
>
RE: Design issue, need feedback
Posted by Ruslan Al-fakikh <ru...@jalent.ru>.
Hey Nerius,
As for the columns number changes - yes, Avro or Thrift can handle that.
As for transforming a value of a row from this 'Age=23' to this '23' - this is what Pig can do for you.
Try something like
b = foreach a generate substring(0,4,$1) AS Age --I haven't tested it, there can be typos
Or maybe some other builtins can do, like regex stuff
Ruslan Al-Fakikh
-----Original Message-----
From: Jonathan Coveney [mailto:jcoveney@gmail.com]
Sent: Tuesday, May 22, 2012 3:47 AM
To: user@pig.apache.org
Subject: Re: Design issue, need feedback
There are a couple of ways that you can do this. One, is that you could make a special loader that converts your format to a map of (key,value) pairs, and then you can project however you want.
Another (better, if at all possible) way would be to use something like Avro or Thrift that allows you to update your schemas in backwards compatible ways. I highly recommend leveraging one of these projects instead of rolling your own janky file format. Starting from nothing Avro would probably be easier, though here at Twitter we have standardized on Thrift. Pig can work with both.
2012/5/21 Nerius Landys <nl...@gmail.com>
> I'm trying to do something with Pig that I believe Pig wasn't really
> designed/intended to handle.
> Normally, the way I'd do things with Pig is by feeding it data like so:
>
> Fred 23
> Adam 25
> Mary 21
>
> ...where the columns are name and age.
>
> Now, a requirement came up to have the Pig script be more flexible
> when the format of the data changes. In fact with the system I'm
> working in, the format of the data will change quite often. Let me
> explain. So now in one scenario the data incoming would be this,
> literally:
>
> Name=Fred Age=23
> Name=Adam Age=25
> Name=Mary Age=21
>
> ...and if the format of the data changes somewhat it may appear like
> this for example:
>
> Age=23 Sex=male Name=Fred
> Age=25 Sex=male Name=Adam
> Age=21 Sex=female Name=Mary
>
> <All data is tab delimited of course.> However in all cases the only
> data I'm interested in is name and age, in that order. And in all
> cases these two pieces of data are guaranteed to appear.
>
> So my question is, how easy and efficient would it be to write
> something in the Pig langauge that transforms the key/value rows into
> the format in my first example with just name and age columns?
>
> That is, I want to transform, using a fixed series of Pig statements,
> this type of row:
>
> Name=Fred Age=23
>
> to this:
>
> Fred 23
>
> And I want the same series of Pig statements to transform this type of row:
>
> Age=23 Sex=male Name=Fred
>
> to this:
>
> Fred 23
>
> How would I go about doing this, and would this be terribly
> inefficient? Is this just not the way Pig was meant to work?
> I can think of a way to do this using a UDF maybe but is there a way
> to do this using builtins?
>
> Thanks, Nerius
>
Re: Design issue, need feedback
Posted by Nerius Landys <nl...@gmail.com>.
> Why not convert your file format into a Map, which is essentially what it
> is (key, value) pairs, and then you can just project out the values that
> you want? It's a little annoying to have to project everything manually,
> but will be a lot more maintainable (and this is essentially what
> JSONStorage does).
Somehow I thought that having 1 map per row of data would be overkill.
Like in, lots of memory overhead. And slower.
Re: Design issue, need feedback
Posted by Jonathan Coveney <jc...@gmail.com>.
Why not convert your file format into a Map, which is essentially what it
is (key, value) pairs, and then you can just project out the values that
you want? It's a little annoying to have to project everything manually,
but will be a lot more maintainable (and this is essentially what
JSONStorage does).
2012/5/22 Nerius Landys <nl...@gmail.com>
> > There are a couple of ways that you can do this. One, is that you could
> > make a special loader that converts your format to a map of (key,value)
> > pairs, and then you can project however you want.
> >
> > Another (better, if at all possible) way would be to use something like
> > Avro or Thrift that allows you to update your schemas in backwards
> > compatible ways. I highly recommend leveraging one of these projects
> > instead of rolling your own janky file format. Starting from nothing Avro
> > would probably be easier, though here at Twitter we have standardized on
> > Thrift. Pig can work with both.
>
> Actually, we already have our own janky file format. We have this
> tightly bit-packed binary format which essentially stores rows of
> information much like the rows going through a Pig process.
>
> And so what I've done prior is already written a LoadFunc for Pig
> which parses the binary data and returns tuples for Pig. I'm now just
> debating on whether to return tuples in the format:
>
> (Fred,23)
>
> ...or tuples in the format
>
> (Name=Fred,Age=23)
>
> I was leaning towards the latter format for my tuples because if the
> data in the binary format changes [in a minor fashion], I would be
> generating for example:
>
> (Age=23,Sex=male,Name=Fred)
>
> ...and I would have a chance at dealing with the change without
> modifying a Pig script, but with the added problem that my Pig script
> would have to do more parsing and interpretation of the key/value
> nature of the elements in the tuples.
>
> So the question really boils down to, what kind of tuples should I
> generate in my LoadFunc to make future maintenance of the Pig scripts
> most easy?
>
> If I stick with the key/value nature, to make the Pig rows easy to
> work with I'd like to have some Pig rules that simplified the
> key/value strings to just the values in a predetermined order. So
> what I was thinking was maybe writing a UDF like so:
>
> Foo($row, 'Name,Age')
>
> ...which would transform a row like this:
>
> (Age=23,Sex=male,Name=Fred)
>
> ...to a row like this:
>
> (Fred,23)
>
> , making it easier to work with later on in the Pig script. If I did
> write a UDF like this, is there a way to pass the entire row ("$row"
> in above example) to the UDF?
>
> Any other advice appreciated. But bottom line is that the file format
> is already defined and I'm just writing the LoadFunc to return tuples.
>
Re: Design issue, need feedback
Posted by Nerius Landys <nl...@gmail.com>.
> There are a couple of ways that you can do this. One, is that you could
> make a special loader that converts your format to a map of (key,value)
> pairs, and then you can project however you want.
>
> Another (better, if at all possible) way would be to use something like
> Avro or Thrift that allows you to update your schemas in backwards
> compatible ways. I highly recommend leveraging one of these projects
> instead of rolling your own janky file format. Starting from nothing Avro
> would probably be easier, though here at Twitter we have standardized on
> Thrift. Pig can work with both.
Actually, we already have our own janky file format. We have this
tightly bit-packed binary format which essentially stores rows of
information much like the rows going through a Pig process.
And so what I've done prior is already written a LoadFunc for Pig
which parses the binary data and returns tuples for Pig. I'm now just
debating on whether to return tuples in the format:
(Fred,23)
...or tuples in the format
(Name=Fred,Age=23)
I was leaning towards the latter format for my tuples because if the
data in the binary format changes [in a minor fashion], I would be
generating for example:
(Age=23,Sex=male,Name=Fred)
...and I would have a chance at dealing with the change without
modifying a Pig script, but with the added problem that my Pig script
would have to do more parsing and interpretation of the key/value
nature of the elements in the tuples.
So the question really boils down to, what kind of tuples should I
generate in my LoadFunc to make future maintenance of the Pig scripts
most easy?
If I stick with the key/value nature, to make the Pig rows easy to
work with I'd like to have some Pig rules that simplified the
key/value strings to just the values in a predetermined order. So
what I was thinking was maybe writing a UDF like so:
Foo($row, 'Name,Age')
...which would transform a row like this:
(Age=23,Sex=male,Name=Fred)
...to a row like this:
(Fred,23)
, making it easier to work with later on in the Pig script. If I did
write a UDF like this, is there a way to pass the entire row ("$row"
in above example) to the UDF?
Any other advice appreciated. But bottom line is that the file format
is already defined and I'm just writing the LoadFunc to return tuples.
Re: Design issue, need feedback
Posted by Jonathan Coveney <jc...@gmail.com>.
There are a couple of ways that you can do this. One, is that you could
make a special loader that converts your format to a map of (key,value)
pairs, and then you can project however you want.
Another (better, if at all possible) way would be to use something like
Avro or Thrift that allows you to update your schemas in backwards
compatible ways. I highly recommend leveraging one of these projects
instead of rolling your own janky file format. Starting from nothing Avro
would probably be easier, though here at Twitter we have standardized on
Thrift. Pig can work with both.
2012/5/21 Nerius Landys <nl...@gmail.com>
> I'm trying to do something with Pig that I believe Pig wasn't really
> designed/intended to handle.
> Normally, the way I'd do things with Pig is by feeding it data like so:
>
> Fred 23
> Adam 25
> Mary 21
>
> ...where the columns are name and age.
>
> Now, a requirement came up to have the Pig script be more flexible
> when the format of the data changes. In fact with the system I'm
> working in, the format of the data will change quite often. Let me
> explain. So now in one scenario the data incoming would be this,
> literally:
>
> Name=Fred Age=23
> Name=Adam Age=25
> Name=Mary Age=21
>
> ...and if the format of the data changes somewhat it may appear like
> this for example:
>
> Age=23 Sex=male Name=Fred
> Age=25 Sex=male Name=Adam
> Age=21 Sex=female Name=Mary
>
> <All data is tab delimited of course.>
> However in all cases the only data I'm interested in is name and age,
> in that order. And in all cases these two pieces of data are
> guaranteed to appear.
>
> So my question is, how easy and efficient would it be to write
> something in the Pig langauge that transforms the key/value rows into
> the format in my first example with just name and age columns?
>
> That is, I want to transform, using a fixed series of Pig statements,
> this type of row:
>
> Name=Fred Age=23
>
> to this:
>
> Fred 23
>
> And I want the same series of Pig statements to transform this type of row:
>
> Age=23 Sex=male Name=Fred
>
> to this:
>
> Fred 23
>
> How would I go about doing this, and would this be terribly
> inefficient? Is this just not the way Pig was meant to work?
> I can think of a way to do this using a UDF maybe but is there a way
> to do this using builtins?
>
> Thanks, Nerius
>