You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Ron Bodkin <rb...@quantcast.com> on 2007/11/07 04:22:37 UTC

Working with Custom Objects

Hi All,

 

We're very interested in using Pig for some of our cluster computing. It's
great that PIG is now an Apache incubator project. However, we are using our
own existing Row file format that we use to hold data values for our
computing jobs. This predates the existence of Hadoop records, but we like
our style, which uses Java reflection with runtime generated bytecode for
efficiency. We are planning to offer our row file format as a Hadoop
contribution in future.

 

In any event, we would really like to have interfaces we could implement to
plug-in our own classes that would provide atoms, tuples and bags in PIG. In
particular, we'd like to be able to support:

*	Typed Atoms that might include numbers or arbitrary objects 
*	Tuples that would typically be mapped to fields we've defined in our
objects (or in some cases arrays inside objects) 
*	Bags that might include collections or arrays of objects using our
own data structures, either in memory or disk-based 

 

If PIG had these data structures represented with interfaces that could be
provided as an external plug-in, we believe it would make it possible to use
for a lot of new applications, including some of the numerically-oriented
ones we need. As the system stands now, it looks like these are implemented
as classes with a number of coupled dependencies in different layers, so it
is hard to add user data types based on a different class hierarchy. We
believe that if we could plugin to these data formats, we could write
operators and I/O functions that would adapt them to work with PIG. I'd be
interested in feedback on this suggestion. We'd be glad to provide some
samples of how we'd like this to work, if there's interest in the idea.

 

Thanks,

Ron

 


RE: Working with Custom Objects

Posted by Ron Bodkin <rb...@quantcast.com>.
Hi Alan,

I see your concern. It probably makes sense to dig more into specific uses
to flesh out the design. I'm not clear how a tuple consisting of one
collection would work. Could you access named members? E.g., if we have a
class like this:

public class MyRow {
    public String name;
    public Integer count;
    public double[] row;
}

public class SummaryRow {
    public String name;
    public Integer count;
    public Double summary;
}

We'd very much like to be able to read in and process with instances of
these kinds of class from a file and operate with them in pig, so we could
say things like this (apologies for any errors in my Pig code):

A = load '/my/dfs/directory' using RowStorage() as com.quantcast.row.MyRow;
B = group A by name;
C = foreach B GENERATE (B.name, SUM(B.count), mySummaryOperator(B)) as
com.quantcast.row.SummaryRow;
store C into '/my/dfs/results';

Where we have defined our own Java-based operator, mySummaryOperator, which
takes in a parameter that's compatible with MyRow

We don't use Java beans style but some people would probably want a facility
like that too, e.g., having

public class TheirBean {
   private String name;
   public String getName() { return name; }
   public void setName(String name) { this.name = name; }
...
}

Would your proposal for container classes support something like that? How
would you see it working?

I'll send a follow up message to discuss how we want to work with bags, so
we can see how it would match up.

Thanks,
Ron

-----Original Message-----
From: Alan Gates [mailto:gates@yahoo-inc.com] 
Sent: Monday, November 12, 2007 10:33 AM
To: pig-user@incubator.apache.org
Subject: Re: Working with Custom Objects

We are definitely open to adding user defined types.  They were left out 
of the current version of the types document because, after some 
discussion, we concluded we did not yet understand user requirements 
well enough to propose anything meaningful.

As far as turning tuples and bags into interfaces, my concern is that 
tuples and bags are tightly integrated into pig.  If users could 
implement their own realizations of tuples and bags, they could end up 
making decisions that made sense for their design but that were not 
optimal for the rest of pig.  I don't want a situation where people 
implement their own tuples and suddenly pig starts going 10x slower.  
Also, I'm concerned that having tuples and bags exposed as interfaces 
would make future changes and optimizations very difficult for pig 
developers and very disruptive for users. 

Would it meet your requirements if, instead of exposing tuples and bags 
as interfaces, pig provided the ability for users to define their own 
container classes as user defined types?  This would allow pig to 
optimize its internal data structures for itself while allowing users to 
define their own collections.  It would probably result in some 
redundant looking data structures (e.g. this is a bag of tuples, each 
with one entry of a user defined collection X).

Alan.

Ron Bodkin wrote:
> Overall, this looks like a good approach to adding types to Pig. We would
> like to allow another type of data atom, an object, which could at least
be
> passed to external functions. This would be a bit like the use of unknown,
> except that the implementation would hold a reference to an arbitrary Java
> object. To allow for flexibility at other levels, it would be important to
> have interfaces so we could have a tuple implementation based on our rows,
> which would know how to get the fields in order as the tuple values.
Having
> this level of flexibility would go a long way to allow integration between
> typed Java code and PIG code, which seems like an important capability.
> Likewise, having interfaces for bags would let us use our own iteration or
> file access approach to managing collections of tuples.
>
> I'd also suggest providing external facilities to provide metadata about
> types, e.g., standard annotations for user-defined functions (or Java
> classes), and a way for file I/O to provide type information from file
> metadata.
>
> Also, on a small note "comparitor" (sic) is misspelled throughout, it
should
> be comparator.
>
> -----Original Message-----
> From: Alan Gates [mailto:gates@yahoo-inc.com] 
> Sent: Thursday, November 08, 2007 9:10 AM
> To: pig-user@incubator.apache.org
> Subject: Re: Working with Custom Objects
>
> Ron, have you taken a look at 
> http://wiki.apache.org/pig/PigTypesFunctionalSpec ?
> It won't address all of your concerns, but it will help you get an idea 
> of what we've been thinking about so far.  Then we can add your feedback 
> to to the doc from there.
>
> Alan.
>
> Ron Bodkin wrote:
>   
>> Hi All,
>>
>>  
>>
>> We're very interested in using Pig for some of our cluster computing.
It's
>> great that PIG is now an Apache incubator project. However, we are using
>>     
> our
>   
>> own existing Row file format that we use to hold data values for our
>> computing jobs. This predates the existence of Hadoop records, but we
like
>> our style, which uses Java reflection with runtime generated bytecode for
>> efficiency. We are planning to offer our row file format as a Hadoop
>> contribution in future.
>>
>>  
>>
>> In any event, we would really like to have interfaces we could implement
>>     
> to
>   
>> plug-in our own classes that would provide atoms, tuples and bags in PIG.
>>     
> In
>   
>> particular, we'd like to be able to support:
>>
>> *	Typed Atoms that might include numbers or arbitrary objects 
>> *	Tuples that would typically be mapped to fields we've defined in our
>> objects (or in some cases arrays inside objects) 
>> *	Bags that might include collections or arrays of objects using our
>> own data structures, either in memory or disk-based 
>>
>>  
>>
>> If PIG had these data structures represented with interfaces that could
be
>> provided as an external plug-in, we believe it would make it possible to
>>     
> use
>   
>> for a lot of new applications, including some of the numerically-oriented
>> ones we need. As the system stands now, it looks like these are
>>     
> implemented
>   
>> as classes with a number of coupled dependencies in different layers, so
>>     
> it
>   
>> is hard to add user data types based on a different class hierarchy. We
>> believe that if we could plugin to these data formats, we could write
>> operators and I/O functions that would adapt them to work with PIG. I'd
be
>> interested in feedback on this suggestion. We'd be glad to provide some
>> samples of how we'd like this to work, if there's interest in the idea.
>>
>>  
>>
>> Thanks,
>>
>> Ron
>>
>>  
>>
>>
>>   
>>     


Re: Working with Custom Objects

Posted by Alan Gates <ga...@yahoo-inc.com>.
We are definitely open to adding user defined types.  They were left out 
of the current version of the types document because, after some 
discussion, we concluded we did not yet understand user requirements 
well enough to propose anything meaningful.

As far as turning tuples and bags into interfaces, my concern is that 
tuples and bags are tightly integrated into pig.  If users could 
implement their own realizations of tuples and bags, they could end up 
making decisions that made sense for their design but that were not 
optimal for the rest of pig.  I don't want a situation where people 
implement their own tuples and suddenly pig starts going 10x slower.  
Also, I'm concerned that having tuples and bags exposed as interfaces 
would make future changes and optimizations very difficult for pig 
developers and very disruptive for users. 

Would it meet your requirements if, instead of exposing tuples and bags 
as interfaces, pig provided the ability for users to define their own 
container classes as user defined types?  This would allow pig to 
optimize its internal data structures for itself while allowing users to 
define their own collections.  It would probably result in some 
redundant looking data structures (e.g. this is a bag of tuples, each 
with one entry of a user defined collection X).

Alan.

Ron Bodkin wrote:
> Overall, this looks like a good approach to adding types to Pig. We would
> like to allow another type of data atom, an object, which could at least be
> passed to external functions. This would be a bit like the use of unknown,
> except that the implementation would hold a reference to an arbitrary Java
> object. To allow for flexibility at other levels, it would be important to
> have interfaces so we could have a tuple implementation based on our rows,
> which would know how to get the fields in order as the tuple values. Having
> this level of flexibility would go a long way to allow integration between
> typed Java code and PIG code, which seems like an important capability.
> Likewise, having interfaces for bags would let us use our own iteration or
> file access approach to managing collections of tuples.
>
> I'd also suggest providing external facilities to provide metadata about
> types, e.g., standard annotations for user-defined functions (or Java
> classes), and a way for file I/O to provide type information from file
> metadata.
>
> Also, on a small note "comparitor" (sic) is misspelled throughout, it should
> be comparator.
>
> -----Original Message-----
> From: Alan Gates [mailto:gates@yahoo-inc.com] 
> Sent: Thursday, November 08, 2007 9:10 AM
> To: pig-user@incubator.apache.org
> Subject: Re: Working with Custom Objects
>
> Ron, have you taken a look at 
> http://wiki.apache.org/pig/PigTypesFunctionalSpec ?
> It won't address all of your concerns, but it will help you get an idea 
> of what we've been thinking about so far.  Then we can add your feedback 
> to to the doc from there.
>
> Alan.
>
> Ron Bodkin wrote:
>   
>> Hi All,
>>
>>  
>>
>> We're very interested in using Pig for some of our cluster computing. It's
>> great that PIG is now an Apache incubator project. However, we are using
>>     
> our
>   
>> own existing Row file format that we use to hold data values for our
>> computing jobs. This predates the existence of Hadoop records, but we like
>> our style, which uses Java reflection with runtime generated bytecode for
>> efficiency. We are planning to offer our row file format as a Hadoop
>> contribution in future.
>>
>>  
>>
>> In any event, we would really like to have interfaces we could implement
>>     
> to
>   
>> plug-in our own classes that would provide atoms, tuples and bags in PIG.
>>     
> In
>   
>> particular, we'd like to be able to support:
>>
>> *	Typed Atoms that might include numbers or arbitrary objects 
>> *	Tuples that would typically be mapped to fields we've defined in our
>> objects (or in some cases arrays inside objects) 
>> *	Bags that might include collections or arrays of objects using our
>> own data structures, either in memory or disk-based 
>>
>>  
>>
>> If PIG had these data structures represented with interfaces that could be
>> provided as an external plug-in, we believe it would make it possible to
>>     
> use
>   
>> for a lot of new applications, including some of the numerically-oriented
>> ones we need. As the system stands now, it looks like these are
>>     
> implemented
>   
>> as classes with a number of coupled dependencies in different layers, so
>>     
> it
>   
>> is hard to add user data types based on a different class hierarchy. We
>> believe that if we could plugin to these data formats, we could write
>> operators and I/O functions that would adapt them to work with PIG. I'd be
>> interested in feedback on this suggestion. We'd be glad to provide some
>> samples of how we'd like this to work, if there's interest in the idea.
>>
>>  
>>
>> Thanks,
>>
>> Ron
>>
>>  
>>
>>
>>   
>>     

RE: Working with Custom Objects

Posted by Ron Bodkin <rb...@quantcast.com>.
Overall, this looks like a good approach to adding types to Pig. We would
like to allow another type of data atom, an object, which could at least be
passed to external functions. This would be a bit like the use of unknown,
except that the implementation would hold a reference to an arbitrary Java
object. To allow for flexibility at other levels, it would be important to
have interfaces so we could have a tuple implementation based on our rows,
which would know how to get the fields in order as the tuple values. Having
this level of flexibility would go a long way to allow integration between
typed Java code and PIG code, which seems like an important capability.
Likewise, having interfaces for bags would let us use our own iteration or
file access approach to managing collections of tuples.

I'd also suggest providing external facilities to provide metadata about
types, e.g., standard annotations for user-defined functions (or Java
classes), and a way for file I/O to provide type information from file
metadata.

Also, on a small note "comparitor" (sic) is misspelled throughout, it should
be comparator.

-----Original Message-----
From: Alan Gates [mailto:gates@yahoo-inc.com] 
Sent: Thursday, November 08, 2007 9:10 AM
To: pig-user@incubator.apache.org
Subject: Re: Working with Custom Objects

Ron, have you taken a look at 
http://wiki.apache.org/pig/PigTypesFunctionalSpec ?
It won't address all of your concerns, but it will help you get an idea 
of what we've been thinking about so far.  Then we can add your feedback 
to to the doc from there.

Alan.

Ron Bodkin wrote:
> Hi All,
>
>  
>
> We're very interested in using Pig for some of our cluster computing. It's
> great that PIG is now an Apache incubator project. However, we are using
our
> own existing Row file format that we use to hold data values for our
> computing jobs. This predates the existence of Hadoop records, but we like
> our style, which uses Java reflection with runtime generated bytecode for
> efficiency. We are planning to offer our row file format as a Hadoop
> contribution in future.
>
>  
>
> In any event, we would really like to have interfaces we could implement
to
> plug-in our own classes that would provide atoms, tuples and bags in PIG.
In
> particular, we'd like to be able to support:
>
> *	Typed Atoms that might include numbers or arbitrary objects 
> *	Tuples that would typically be mapped to fields we've defined in our
> objects (or in some cases arrays inside objects) 
> *	Bags that might include collections or arrays of objects using our
> own data structures, either in memory or disk-based 
>
>  
>
> If PIG had these data structures represented with interfaces that could be
> provided as an external plug-in, we believe it would make it possible to
use
> for a lot of new applications, including some of the numerically-oriented
> ones we need. As the system stands now, it looks like these are
implemented
> as classes with a number of coupled dependencies in different layers, so
it
> is hard to add user data types based on a different class hierarchy. We
> believe that if we could plugin to these data formats, we could write
> operators and I/O functions that would adapt them to work with PIG. I'd be
> interested in feedback on this suggestion. We'd be glad to provide some
> samples of how we'd like this to work, if there's interest in the idea.
>
>  
>
> Thanks,
>
> Ron
>
>  
>
>
>   

Re: Working with Custom Objects

Posted by Alan Gates <ga...@yahoo-inc.com>.
Ron, have you taken a look at 
http://wiki.apache.org/pig/PigTypesFunctionalSpec ?
It won't address all of your concerns, but it will help you get an idea 
of what we've been thinking about so far.  Then we can add your feedback 
to to the doc from there.

Alan.

Ron Bodkin wrote:
> Hi All,
>
>  
>
> We're very interested in using Pig for some of our cluster computing. It's
> great that PIG is now an Apache incubator project. However, we are using our
> own existing Row file format that we use to hold data values for our
> computing jobs. This predates the existence of Hadoop records, but we like
> our style, which uses Java reflection with runtime generated bytecode for
> efficiency. We are planning to offer our row file format as a Hadoop
> contribution in future.
>
>  
>
> In any event, we would really like to have interfaces we could implement to
> plug-in our own classes that would provide atoms, tuples and bags in PIG. In
> particular, we'd like to be able to support:
>
> *	Typed Atoms that might include numbers or arbitrary objects 
> *	Tuples that would typically be mapped to fields we've defined in our
> objects (or in some cases arrays inside objects) 
> *	Bags that might include collections or arrays of objects using our
> own data structures, either in memory or disk-based 
>
>  
>
> If PIG had these data structures represented with interfaces that could be
> provided as an external plug-in, we believe it would make it possible to use
> for a lot of new applications, including some of the numerically-oriented
> ones we need. As the system stands now, it looks like these are implemented
> as classes with a number of coupled dependencies in different layers, so it
> is hard to add user data types based on a different class hierarchy. We
> believe that if we could plugin to these data formats, we could write
> operators and I/O functions that would adapt them to work with PIG. I'd be
> interested in feedback on this suggestion. We'd be glad to provide some
> samples of how we'd like this to work, if there's interest in the idea.
>
>  
>
> Thanks,
>
> Ron
>
>  
>
>
>