You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by "Ankur C. Goel" <ga...@yahoo-inc.com> on 2010/02/22 08:41:40 UTC
Algorithm implementations in Pig
Hi Folks,
I would like to know how mahout community feels about having some of the Mahout algorithms implemented in pig - http://hadoop.apache.org/pig. The benefits of using Pig are many including.
1. Small learning curve, people with a bit of SQL knowledge will find it very easy.
2. Operations like grouping, aggregations, join need just few lines of pig code.
3. Insulation against Hadoop complexity - Job chains and JobConf.
4. Quick prototyping and hence increased programmer productivity.
I had Sean's opinion on this and he was not too comfortable with the Idea of having things in different languages in Mahout. However, given the benefits of PIG, I feel otherwise. I may be biased here due to my own experience of being able to do more in lesser time in Pig then in M/R, so I thought let me ask how folks feel.
Ted, I believe you have some PIG experience yourself so any thoughts on this ?
Regards
-@nkur
Re: Algorithm implementations in Pig
Posted by David Stuart <da...@progressivealliance.co.uk>.
Seems like the guys at twitter are going down the pig/hadoop http://highscalability.com/blog/2010/2/19/twitters-plan-to-analyze-100-billion-tweets.html route could be worth getting them on board the Mahout wagon especially with previous discussion had about classification efforts http://old.nabble.com/Twitter-Classification-td27227638.html
On 22 Feb 2010, at 12:13, Grant Ingersoll wrote:
> I'm all for Pig, especially once we are a TLP. I haven't had the proper time to review the PLSI implementation, but it looks useful. I agree on the other points, though, in that I think we it would be nice to have consistent formats based on Vector so that things can be more portable.
>
>
> On Feb 22, 2010, at 2:41 AM, Ankur C. Goel wrote:
>
>> Hi Folks,
>> I would like to know how mahout community feels about having some of the Mahout algorithms implemented in pig - http://hadoop.apache.org/pig. The benefits of using Pig are many including.
>>
>>
>> 1. Small learning curve, people with a bit of SQL knowledge will find it very easy.
>> 2. Operations like grouping, aggregations, join need just few lines of pig code.
>> 3. Insulation against Hadoop complexity - Job chains and JobConf.
>> 4. Quick prototyping and hence increased programmer productivity.
>>
>> I had Sean's opinion on this and he was not too comfortable with the Idea of having things in different languages in Mahout. However, given the benefits of PIG, I feel otherwise. I may be biased here due to my own experience of being able to do more in lesser time in Pig then in M/R, so I thought let me ask how folks feel.
>>
>> Ted, I believe you have some PIG experience yourself so any thoughts on this ?
>>
>> Regards
>> -@nkur
>
Re: Algorithm implementations in Pig
Posted by Grant Ingersoll <gs...@apache.org>.
I'm all for Pig, especially once we are a TLP. I haven't had the proper time to review the PLSI implementation, but it looks useful. I agree on the other points, though, in that I think we it would be nice to have consistent formats based on Vector so that things can be more portable.
On Feb 22, 2010, at 2:41 AM, Ankur C. Goel wrote:
> Hi Folks,
> I would like to know how mahout community feels about having some of the Mahout algorithms implemented in pig - http://hadoop.apache.org/pig. The benefits of using Pig are many including.
>
>
> 1. Small learning curve, people with a bit of SQL knowledge will find it very easy.
> 2. Operations like grouping, aggregations, join need just few lines of pig code.
> 3. Insulation against Hadoop complexity - Job chains and JobConf.
> 4. Quick prototyping and hence increased programmer productivity.
>
> I had Sean's opinion on this and he was not too comfortable with the Idea of having things in different languages in Mahout. However, given the benefits of PIG, I feel otherwise. I may be biased here due to my own experience of being able to do more in lesser time in Pig then in M/R, so I thought let me ask how folks feel.
>
> Ted, I believe you have some PIG experience yourself so any thoughts on this ?
>
> Regards
> -@nkur
Re: Algorithm implementations in Pig
Posted by Robin Anil <ro...@gmail.com>.
On Mon, Feb 22, 2010 at 1:55 PM, Ted Dunning <te...@gmail.com> wrote:
> I see pig as useful for data preparation, but for any numerical tasks, it
> is
> likely to be completely hopeless.
>
PIG will be a great tool to experiment quickly on algorithms. But, with
people here trying to focus on using Vector to standardize the input output
process, It will be tough for the small bunch here to port that to PIG, or
help PIG scripts reuse it. As long as the input output of PIG based
algorithmns is based on VectorWritable, I dont see any problem not including
PIG. But bear in mind the previous PIG submission
https://issues.apache.org/jira/browse/MAHOUT-106 still haven't moved in to
the trunk. If anyone is willing to help standardize on using PIG with
vectors as input they are more than welcome.
One thing we definitely dont want to do at this point is for all algorithms
to have all different kinds of input format.
Robin
Re: Algorithm implementations in Pig
Posted by "Ankur C. Goel" <ga...@yahoo-inc.com>.
In the next pig release (0.7) Pig's load/store func would be moving to use hadoop's input/output format. So there are some changes planned for that - http://wiki.apache.org/pig/Pig070IncompatibleChanges
After that I don't expect any interface level change in UDF.
-@nkur
On 2/22/10 10:10 PM, "Ted Dunning" <te...@gmail.com> wrote:
Has the interface for writing UDF's stabilized? For quite some time, the
UDF API was changing every 3 months.
On Mon, Feb 22, 2010 at 12:35 AM, Jeff Zhang <zj...@gmail.com> wrote:
> Pig can only make the implementation of map-reduce easier, the numerical
> computation can been done in UDF.
>
--
Ted Dunning, CTO
DeepDyve
Re: Algorithm implementations in Pig
Posted by Ted Dunning <te...@gmail.com>.
Has the interface for writing UDF's stabilized? For quite some time, the
UDF API was changing every 3 months.
On Mon, Feb 22, 2010 at 12:35 AM, Jeff Zhang <zj...@gmail.com> wrote:
> Pig can only make the implementation of map-reduce easier, the numerical
> computation can been done in UDF.
>
--
Ted Dunning, CTO
DeepDyve
Re: Algorithm implementations in Pig
Posted by Jeff Zhang <zj...@gmail.com>.
Pig can only make the implementation of map-reduce easier, the numerical
computation can been done in UDF. And piglet is a DSL upon pig latin which
make pig support loop.
http://github.com/iconara/piglet
On Mon, Feb 22, 2010 at 4:25 PM, Ted Dunning <te...@gmail.com> wrote:
> I see pig as useful for data preparation, but for any numerical tasks, it
> is
> likely to be completely hopeless.
>
> On Mon, Feb 22, 2010 at 12:16 AM, Jeff Zhang <zj...@gmail.com> wrote:
>
> >
> > Glad to hear here that mahout devs are interested in pig. Actually I
> > believe
> > pig is very helpful when you want to quickly implement a prototype of
> > machine learning algorithms. And Pig has java API, it is easy to
> integrate
> > pig script with java. Maybe we can start with implementing NB using pig
> > first.
>
>
>
>
> --
> Ted Dunning, CTO
> DeepDyve
>
--
Best Regards
Jeff Zhang
Re: Algorithm implementations in Pig
Posted by Ted Dunning <te...@gmail.com>.
I see pig as useful for data preparation, but for any numerical tasks, it is
likely to be completely hopeless.
On Mon, Feb 22, 2010 at 12:16 AM, Jeff Zhang <zj...@gmail.com> wrote:
>
> Glad to hear here that mahout devs are interested in pig. Actually I
> believe
> pig is very helpful when you want to quickly implement a prototype of
> machine learning algorithms. And Pig has java API, it is easy to integrate
> pig script with java. Maybe we can start with implementing NB using pig
> first.
--
Ted Dunning, CTO
DeepDyve
Re: Algorithm implementations in Pig
Posted by Jeff Zhang <zj...@gmail.com>.
Hi,
Glad to hear here that mahout devs are interested in pig. Actually I believe
pig is very helpful when you want to quickly implement a prototype of
machine learning algorithms. And Pig has java API, it is easy to integrate
pig script with java. Maybe we can start with implementing NB using pig
first.
On Mon, Feb 22, 2010 at 3:56 PM, Ted Dunning <te...@gmail.com> wrote:
> I have had both positive and negative results with PIG.
>
> The positive results were that I was able to express large recommendation
> computations in a very concise way. That was really helpful.
>
> My negative results have been to do with the brittle nature of PIG vis a
> vis
> the version of the underlying hadoop system. That problem may have abated
> somewhat as everybody in the world except me and Amazon's EMR has pretty
> much piled up on version 20. I also know little about how Pig would
> interface well with other components. I know that I have had difficulty in
> the past injecting outside information into Pig, but that has been
> improved. I also know that "Pigs eat anything", but have no clear idea how
> well this would play out with, say, our vector formats and vectorizers.
>
> Ankur, what recent experience do you have? How well do PIG scripts play
> with other programs any more?
>
> On Sun, Feb 21, 2010 at 11:41 PM, Ankur C. Goel <gankur@yahoo-inc.com
> >wrote:
>
> > I had Sean's opinion on this and he was not too comfortable with the Idea
> > of having things in different languages in Mahout. However, given the
> > benefits of PIG, I feel otherwise. I may be biased here due to my own
> > experience of being able to do more in lesser time in Pig then in M/R,
> so I
> > thought let me ask how folks feel.
> >
> > Ted, I believe you have some PIG experience yourself so any thoughts on
> > this ?
> >
>
>
>
> --
> Ted Dunning, CTO
> DeepDyve
>
--
Best Regards
Jeff Zhang
Re: Algorithm implementations in Pig
Posted by Ted Dunning <te...@gmail.com>.
Good answer.
On Mon, Feb 22, 2010 at 8:52 PM, Ankur C. Goel <ga...@yahoo-inc.com> wrote:
> Those would be passed as parameters either through -param option or through
> a parameter file with -param_file option and the pig's preprocessor just
> substitutes the values in your script.
> Since its just a blind parameter substitution, in my shingling script I
> even had the schema definition passed to it. I suppose passing input field
> name shouldn't be an issue as long as it is valid
> In the context of script execution plan.
>
>
Re: Algorithm implementations in Pig
Posted by "Ankur C. Goel" <ga...@yahoo-inc.com>.
Those would be passed as parameters either through -param option or through a parameter file with -param_file option and the pig's preprocessor just substitutes the values in your script.
Since its just a blind parameter substitution, in my shingling script I even had the schema definition passed to it. I suppose passing input field name shouldn't be an issue as long as it is valid
In the context of script execution plan.
-@nkur
10 11:32 PM, "Ted Dunning" <te...@gmail.com> wrote:
As an interesting test case, can you write a pig program that counts words.
BUT, it takes an input file name AND an input field name.
On Mon, Feb 22, 2010 at 9:56 AM, Ted Dunning <te...@gmail.com> wrote:
>
> That isn't an issue here. It is the invocation of pig programs and passing
> useful information to them that is the problem.
>
>
> On Mon, Feb 22, 2010 at 9:20 AM, Ankur C. Goel <ga...@yahoo-inc.com>wrote:
>
>> Scripting ability while still limited has better streaming support so you
>> can have relations streamed
>> Into a custom script executing in either map or reduce phase depending
>> upon where it is placed.
>>
>
>
>
> --
> Ted Dunning, CTO
> DeepDyve
>
>
--
Ted Dunning, CTO
DeepDyve
Re: Algorithm implementations in Pig
Posted by Ted Dunning <te...@gmail.com>.
Indeed. I have observed Pig running considerably faster than hand-written
MR programs, precisely because it is willing and able to do optimizations
that decrease the number of passes over the data. These optimizations break
abstraction boundaries in a way that would be very unpleasant or infeasible
maintenance-wise in the hand-written Java code.
In general, I don't think performance will be all that strongly impacted by
introducing pig components.
On Tue, Feb 23, 2010 at 11:54 PM, Ankur C. Goel <ga...@yahoo-inc.com>wrote:
> Pig also provides very nice features like MultiQuery optimization and
> skewed & merge join that are hard to implement in Java M/R every time you
> need them.
>
> With the latest pig release 0.6 the performance gap between Java M/R and
> Pig has been narrowed to a good extent.
>
--
Ted Dunning, CTO
DeepDyve
Re: Algorithm implementations in Pig
Posted by "Ankur C. Goel" <ga...@yahoo-inc.com>.
Hey Pallavi,
Check this out - http://wiki.apache.org/pig/PigMix. The performance is consistently improving
-@nkur
On 2/24/10 2:13 PM, "Palleti, Pallavi" <pa...@corp.aol.com> wrote:
Thanks for the clarification Ankur. Do you have any performance
comparison between pig-0.6 and Hadoop? I would be interested to look at
the same. Last I heard about the comparison was in
http://osdir.com/ml/hive-user-hadoop-apache/2009-06/msg00078.html.
Pig-0.7.0 seems interesting. Thanks for sharing the information. I am
looking forward to experiment with it.
Thanks
Pallavi
-----Original Message-----
From: Ankur C. Goel [mailto:gankur@yahoo-inc.com]
Sent: Wednesday, February 24, 2010 1:24 PM
To: mahout-dev@lucene.apache.org
Subject: Re: Algorithm implementations in Pig
Pallavi,
Thanks for your comments. Some clarifications w.r.t pig.
Pig does not generate any M/R code. What is it generates is logical,
physical and map-reduce plans that are nothing but DAGs. The map-reduce
plan is then interpreted by pig's own mappers/reducers. The plan
generation itself is done on the client side and takes few seconds or
minutes (if you have a really big script).
About performance tuning in hadoop, all the M/R parameters can be
adjusted in pig to have the same effect they'd have in Java M/R
programs. Pig 0.7 is moving towards using hadoop's input/output format
in its load/store functions, so your custom i/o formats can be easily
reused with little additional effort.
Pig also provides very nice features like MultiQuery optimization and
skewed & merge join that are hard to implement in Java M/R every time
you need them.
With the latest pig release 0.6 the performance gap between Java M/R and
Pig has been narrowed to a good extent.
Simple statistical measures that you would use to understand or
preprocess your data are very easy to do with just few lines of pig code
and lot of utility UDFs are available for that.
Besides all the good things, I agree that there are compatibility issues
running pig-x on hadoop-y but this has also to do with new features of
Hadoop that pig is able to exploit in its pipeline.
I also agree with the general opinion that for Pig's adoption in Mahout
land it should play out well with Mahout's vector formats.
At the moment I don't have the proper free time to look into this but
will surely get back to evaluating the feasibility of this integration
in the coming few weeks. Till then any of the interested folks can fork
a JIRA for this and work on it.
On 2/24/10 12:27 PM, "Palleti, Pallavi" <pa...@corp.aol.com>
wrote:
I too have mixed opinion w.r.t pig. Pig would be a good choice to
quickly prototype and test. However, following are the pitfalls I have
observed in pig.
It is not easy to debug in pig. Also, it have performance issues as it
is a layer on top of hadoop, so the overhead of converting pig into
map-reduce code. Also, when the code is available in hadoop, it is in
developer/user's hand to improve the performance by using various
parameters say, no of mappers, different input formats, etc. However is
not the case with pig. Also,there are some compatibility issues with pig
and hadoop. Say, if I am using pig-x version on hadoop-y version, there
might be some compatibility issues and need to spend time on resolving
the same as it is not easy to figure out the errors.
I believe the main motto of mahout is to propose scalable algorithms
which can be used to solve some real world problems. In such case, if
pig has got rid of above pitfalls, then it would be good choice as we
will have very less developing time efforts.
Thanks
Pallavi
-----Original Message-----
From: Ted Dunning [mailto:ted.dunning@gmail.com]
Sent: Monday, February 22, 2010 11:32 PM
To: mahout-dev@lucene.apache.org
Subject: Re: Algorithm implementations in Pig
As an interesting test case, can you write a pig program that counts
words.
BUT, it takes an input file name AND an input field name.
On Mon, Feb 22, 2010 at 9:56 AM, Ted Dunning <te...@gmail.com>
wrote:
>
> That isn't an issue here. It is the invocation of pig programs and
> passing useful information to them that is the problem.
>
>
> On Mon, Feb 22, 2010 at 9:20 AM, Ankur C. Goel
<ga...@yahoo-inc.com>wrote:
>
>> Scripting ability while still limited has better streaming support so
>> you can have relations streamed Into a custom script executing in
>> either map or reduce phase depending upon where it is placed.
>>
>
>
>
> --
> Ted Dunning, CTO
> DeepDyve
>
>
--
Ted Dunning, CTO
DeepDyve
RE: Algorithm implementations in Pig
Posted by "Palleti, Pallavi" <pa...@corp.aol.com>.
Thanks for the clarification Ankur. Do you have any performance
comparison between pig-0.6 and Hadoop? I would be interested to look at
the same. Last I heard about the comparison was in
http://osdir.com/ml/hive-user-hadoop-apache/2009-06/msg00078.html.
Pig-0.7.0 seems interesting. Thanks for sharing the information. I am
looking forward to experiment with it.
Thanks
Pallavi
-----Original Message-----
From: Ankur C. Goel [mailto:gankur@yahoo-inc.com]
Sent: Wednesday, February 24, 2010 1:24 PM
To: mahout-dev@lucene.apache.org
Subject: Re: Algorithm implementations in Pig
Pallavi,
Thanks for your comments. Some clarifications w.r.t pig.
Pig does not generate any M/R code. What is it generates is logical,
physical and map-reduce plans that are nothing but DAGs. The map-reduce
plan is then interpreted by pig's own mappers/reducers. The plan
generation itself is done on the client side and takes few seconds or
minutes (if you have a really big script).
About performance tuning in hadoop, all the M/R parameters can be
adjusted in pig to have the same effect they'd have in Java M/R
programs. Pig 0.7 is moving towards using hadoop's input/output format
in its load/store functions, so your custom i/o formats can be easily
reused with little additional effort.
Pig also provides very nice features like MultiQuery optimization and
skewed & merge join that are hard to implement in Java M/R every time
you need them.
With the latest pig release 0.6 the performance gap between Java M/R and
Pig has been narrowed to a good extent.
Simple statistical measures that you would use to understand or
preprocess your data are very easy to do with just few lines of pig code
and lot of utility UDFs are available for that.
Besides all the good things, I agree that there are compatibility issues
running pig-x on hadoop-y but this has also to do with new features of
Hadoop that pig is able to exploit in its pipeline.
I also agree with the general opinion that for Pig's adoption in Mahout
land it should play out well with Mahout's vector formats.
At the moment I don't have the proper free time to look into this but
will surely get back to evaluating the feasibility of this integration
in the coming few weeks. Till then any of the interested folks can fork
a JIRA for this and work on it.
On 2/24/10 12:27 PM, "Palleti, Pallavi" <pa...@corp.aol.com>
wrote:
I too have mixed opinion w.r.t pig. Pig would be a good choice to
quickly prototype and test. However, following are the pitfalls I have
observed in pig.
It is not easy to debug in pig. Also, it have performance issues as it
is a layer on top of hadoop, so the overhead of converting pig into
map-reduce code. Also, when the code is available in hadoop, it is in
developer/user's hand to improve the performance by using various
parameters say, no of mappers, different input formats, etc. However is
not the case with pig. Also,there are some compatibility issues with pig
and hadoop. Say, if I am using pig-x version on hadoop-y version, there
might be some compatibility issues and need to spend time on resolving
the same as it is not easy to figure out the errors.
I believe the main motto of mahout is to propose scalable algorithms
which can be used to solve some real world problems. In such case, if
pig has got rid of above pitfalls, then it would be good choice as we
will have very less developing time efforts.
Thanks
Pallavi
-----Original Message-----
From: Ted Dunning [mailto:ted.dunning@gmail.com]
Sent: Monday, February 22, 2010 11:32 PM
To: mahout-dev@lucene.apache.org
Subject: Re: Algorithm implementations in Pig
As an interesting test case, can you write a pig program that counts
words.
BUT, it takes an input file name AND an input field name.
On Mon, Feb 22, 2010 at 9:56 AM, Ted Dunning <te...@gmail.com>
wrote:
>
> That isn't an issue here. It is the invocation of pig programs and
> passing useful information to them that is the problem.
>
>
> On Mon, Feb 22, 2010 at 9:20 AM, Ankur C. Goel
<ga...@yahoo-inc.com>wrote:
>
>> Scripting ability while still limited has better streaming support so
>> you can have relations streamed Into a custom script executing in
>> either map or reduce phase depending upon where it is placed.
>>
>
>
>
> --
> Ted Dunning, CTO
> DeepDyve
>
>
--
Ted Dunning, CTO
DeepDyve
Re: Algorithm implementations in Pig
Posted by "Ankur C. Goel" <ga...@yahoo-inc.com>.
Pallavi,
Thanks for your comments. Some clarifications w.r.t pig.
Pig does not generate any M/R code. What is it generates is logical, physical and map-reduce plans that are nothing but DAGs. The map-reduce plan is then interpreted by pig's own mappers/reducers. The plan generation itself is done on the client side and takes few seconds or minutes (if you have a really big script).
About performance tuning in hadoop, all the M/R parameters can be adjusted in pig to have the same effect they'd have in Java M/R programs. Pig 0.7 is moving towards using hadoop's input/output format in its load/store functions, so your custom i/o formats can be easily reused with little additional effort.
Pig also provides very nice features like MultiQuery optimization and skewed & merge join that are hard to implement in Java M/R every time you need them.
With the latest pig release 0.6 the performance gap between Java M/R and Pig has been narrowed to a good extent.
Simple statistical measures that you would use to understand or preprocess your data are very easy to do with just few lines of pig code and lot of utility UDFs are available for that.
Besides all the good things, I agree that there are compatibility issues running pig-x on hadoop-y but this has also to do with new features of Hadoop that pig is able to exploit in its pipeline.
I also agree with the general opinion that for Pig's adoption in Mahout land it should play out well with Mahout's vector formats.
At the moment I don't have the proper free time to look into this but will surely get back to evaluating the feasibility of this integration in the coming few weeks. Till then any of the interested folks can fork a JIRA for this and work on it.
On 2/24/10 12:27 PM, "Palleti, Pallavi" <pa...@corp.aol.com> wrote:
I too have mixed opinion w.r.t pig. Pig would be a good choice to
quickly prototype and test. However, following are the pitfalls I have
observed in pig.
It is not easy to debug in pig. Also, it have performance issues as it
is a layer on top of hadoop, so the overhead of converting pig into
map-reduce code. Also, when the code is available in hadoop, it is in
developer/user's hand to improve the performance by using various
parameters say, no of mappers, different input formats, etc. However is
not the case with pig. Also,there are some compatibility issues with pig
and hadoop. Say, if I am using pig-x version on hadoop-y version, there
might be some compatibility issues and need to spend time on resolving
the same as it is not easy to figure out the errors.
I believe the main motto of mahout is to propose scalable algorithms
which can be used to solve some real world problems. In such case, if
pig has got rid of above pitfalls, then it would be good choice as we
will have very less developing time efforts.
Thanks
Pallavi
-----Original Message-----
From: Ted Dunning [mailto:ted.dunning@gmail.com]
Sent: Monday, February 22, 2010 11:32 PM
To: mahout-dev@lucene.apache.org
Subject: Re: Algorithm implementations in Pig
As an interesting test case, can you write a pig program that counts
words.
BUT, it takes an input file name AND an input field name.
On Mon, Feb 22, 2010 at 9:56 AM, Ted Dunning <te...@gmail.com>
wrote:
>
> That isn't an issue here. It is the invocation of pig programs and
> passing useful information to them that is the problem.
>
>
> On Mon, Feb 22, 2010 at 9:20 AM, Ankur C. Goel
<ga...@yahoo-inc.com>wrote:
>
>> Scripting ability while still limited has better streaming support so
>> you can have relations streamed Into a custom script executing in
>> either map or reduce phase depending upon where it is placed.
>>
>
>
>
> --
> Ted Dunning, CTO
> DeepDyve
>
>
--
Ted Dunning, CTO
DeepDyve
RE: Algorithm implementations in Pig
Posted by "Palleti, Pallavi" <pa...@corp.aol.com>.
I too have mixed opinion w.r.t pig. Pig would be a good choice to
quickly prototype and test. However, following are the pitfalls I have
observed in pig.
It is not easy to debug in pig. Also, it have performance issues as it
is a layer on top of hadoop, so the overhead of converting pig into
map-reduce code. Also, when the code is available in hadoop, it is in
developer/user's hand to improve the performance by using various
parameters say, no of mappers, different input formats, etc. However is
not the case with pig. Also,there are some compatibility issues with pig
and hadoop. Say, if I am using pig-x version on hadoop-y version, there
might be some compatibility issues and need to spend time on resolving
the same as it is not easy to figure out the errors.
I believe the main motto of mahout is to propose scalable algorithms
which can be used to solve some real world problems. In such case, if
pig has got rid of above pitfalls, then it would be good choice as we
will have very less developing time efforts.
Thanks
Pallavi
-----Original Message-----
From: Ted Dunning [mailto:ted.dunning@gmail.com]
Sent: Monday, February 22, 2010 11:32 PM
To: mahout-dev@lucene.apache.org
Subject: Re: Algorithm implementations in Pig
As an interesting test case, can you write a pig program that counts
words.
BUT, it takes an input file name AND an input field name.
On Mon, Feb 22, 2010 at 9:56 AM, Ted Dunning <te...@gmail.com>
wrote:
>
> That isn't an issue here. It is the invocation of pig programs and
> passing useful information to them that is the problem.
>
>
> On Mon, Feb 22, 2010 at 9:20 AM, Ankur C. Goel
<ga...@yahoo-inc.com>wrote:
>
>> Scripting ability while still limited has better streaming support so
>> you can have relations streamed Into a custom script executing in
>> either map or reduce phase depending upon where it is placed.
>>
>
>
>
> --
> Ted Dunning, CTO
> DeepDyve
>
>
--
Ted Dunning, CTO
DeepDyve
Re: Algorithm implementations in Pig
Posted by Ted Dunning <te...@gmail.com>.
As an interesting test case, can you write a pig program that counts words.
BUT, it takes an input file name AND an input field name.
On Mon, Feb 22, 2010 at 9:56 AM, Ted Dunning <te...@gmail.com> wrote:
>
> That isn't an issue here. It is the invocation of pig programs and passing
> useful information to them that is the problem.
>
>
> On Mon, Feb 22, 2010 at 9:20 AM, Ankur C. Goel <ga...@yahoo-inc.com>wrote:
>
>> Scripting ability while still limited has better streaming support so you
>> can have relations streamed
>> Into a custom script executing in either map or reduce phase depending
>> upon where it is placed.
>>
>
>
>
> --
> Ted Dunning, CTO
> DeepDyve
>
>
--
Ted Dunning, CTO
DeepDyve
Re: Algorithm implementations in Pig
Posted by Ted Dunning <te...@gmail.com>.
That isn't an issue here. It is the invocation of pig programs and passing
useful information to them that is the problem.
On Mon, Feb 22, 2010 at 9:20 AM, Ankur C. Goel <ga...@yahoo-inc.com> wrote:
> Scripting ability while still limited has better streaming support so you
> can have relations streamed
> Into a custom script executing in either map or reduce phase depending upon
> where it is placed.
>
--
Ted Dunning, CTO
DeepDyve
Re: Algorithm implementations in Pig
Posted by "Ankur C. Goel" <ga...@yahoo-inc.com>.
I agree with you and while some of that has been remedied, I wouldn't say things are perfect.
Scripting ability while still limited has better streaming support so you can have relations streamed
Into a custom script executing in either map or reduce phase depending upon where it is placed.
If you want to glue together a bunch of map-reduce programs and pig script then the best option is to
Invoke pig from your java program that also manages your M/R chain. Hadoop workflow system (Oozie)
Is coming along which should make this better.
For gluing together multiple pig programs the best there is "exec script.pig" which can be called from inside
your script. However it is not a very neat solution since you would want to pass a bunch of things to the invoked
script and also check for certain conditions to exists. So again a java program or a perl/python/ruby script managing
your chain is a better option.
Regards
-@nkur
On 2/22/10 10:08 PM, "Ted Dunning" <te...@gmail.com> wrote:
Actually, no.
I meant other programs written in pure Java. It used to be that the very
restricted scripting ability of Pig made processing chains composed of Pig
and map-reduce programs very brittle. In fact, just gluing together
multiple Pig programs used to be very ugly.
On Mon, Feb 22, 2010 at 12:42 AM, Ankur C. Goel <ga...@yahoo-inc.com>wrote:
> About interfacing with other systems I assume you have an RDBMS in mind.
--
Ted Dunning, CTO
DeepDyve
Re: Algorithm implementations in Pig
Posted by Ted Dunning <te...@gmail.com>.
Actually, no.
I meant other programs written in pure Java. It used to be that the very
restricted scripting ability of Pig made processing chains composed of Pig
and map-reduce programs very brittle. In fact, just gluing together
multiple Pig programs used to be very ugly.
On Mon, Feb 22, 2010 at 12:42 AM, Ankur C. Goel <ga...@yahoo-inc.com>wrote:
> About interfacing with other systems I assume you have an RDBMS in mind.
--
Ted Dunning, CTO
DeepDyve
Re: Algorithm implementations in Pig
Posted by "Ankur C. Goel" <ga...@yahoo-inc.com>.
Ted,
The latest pig release 0.6.0 on hadoop 20 is a clear winner not just for performance but also for doing a better job of managing memory in its MR job pipeline. Also support for both inner and outer skewed join is something that I found indispensable when dealing with really large datasets. There is support for streaming in pig that lets you stream your relation through an external perl/python/ruby... Script. Also support for UDFs in scripting language is expected in the near future.
About interfacing with other systems I assume you have an RDBMS in mind. There is a patch (for pig 0.7) that lets you write directly from PIG to an RDBMS like MySQL. Support for writing directly to Hbase was always there and has been improved I believe. With 0.7 release pig has decided to let its load/store functions rely on hadoop's input/output format so our vector format shouldn't be a problem IMHO. The only thing I am concerned about is the "not too efficient" Tuple implementation in pig which does not give performance equivalent to Java MR.
Recently I implemented shingling in Pig and found it to work beautifully. One problem that I hit had to too with using clusters to generate recommendations since some clusters were quite large (> 10 K). For this I needed to do a self-join and wanted the join load to be split evenly. That's where skewed join came to the rescue.
Apart from this I also want to contribute my implementation to Mahout (the reason for starting this thread :-))
-@nkur
On 2/22/10 1:26 PM, "Ted Dunning" <te...@gmail.com> wrote:
I have had both positive and negative results with PIG.
The positive results were that I was able to express large recommendation
computations in a very concise way. That was really helpful.
My negative results have been to do with the brittle nature of PIG vis a vis
the version of the underlying hadoop system. That problem may have abated
somewhat as everybody in the world except me and Amazon's EMR has pretty
much piled up on version 20. I also know little about how Pig would
interface well with other components. I know that I have had difficulty in
the past injecting outside information into Pig, but that has been
improved. I also know that "Pigs eat anything", but have no clear idea how
well this would play out with, say, our vector formats and vectorizers.
Ankur, what recent experience do you have? How well do PIG scripts play
with other programs any more?
On Sun, Feb 21, 2010 at 11:41 PM, Ankur C. Goel <ga...@yahoo-inc.com>wrote:
> I had Sean's opinion on this and he was not too comfortable with the Idea
> of having things in different languages in Mahout. However, given the
> benefits of PIG, I feel otherwise. I may be biased here due to my own
> experience of being able to do more in lesser time in Pig then in M/R, so I
> thought let me ask how folks feel.
>
> Ted, I believe you have some PIG experience yourself so any thoughts on
> this ?
>
--
Ted Dunning, CTO
DeepDyve
Re: Algorithm implementations in Pig
Posted by Ted Dunning <te...@gmail.com>.
I have had both positive and negative results with PIG.
The positive results were that I was able to express large recommendation
computations in a very concise way. That was really helpful.
My negative results have been to do with the brittle nature of PIG vis a vis
the version of the underlying hadoop system. That problem may have abated
somewhat as everybody in the world except me and Amazon's EMR has pretty
much piled up on version 20. I also know little about how Pig would
interface well with other components. I know that I have had difficulty in
the past injecting outside information into Pig, but that has been
improved. I also know that "Pigs eat anything", but have no clear idea how
well this would play out with, say, our vector formats and vectorizers.
Ankur, what recent experience do you have? How well do PIG scripts play
with other programs any more?
On Sun, Feb 21, 2010 at 11:41 PM, Ankur C. Goel <ga...@yahoo-inc.com>wrote:
> I had Sean's opinion on this and he was not too comfortable with the Idea
> of having things in different languages in Mahout. However, given the
> benefits of PIG, I feel otherwise. I may be biased here due to my own
> experience of being able to do more in lesser time in Pig then in M/R, so I
> thought let me ask how folks feel.
>
> Ted, I believe you have some PIG experience yourself so any thoughts on
> this ?
>
--
Ted Dunning, CTO
DeepDyve